2024-05-21T17:08:09.739 INFO:root:teuthology version: 0.0.1.dev287+gb9e3da8 2024-05-21T17:08:09.739 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} --archive /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922 --owner scheduled_yuriw@teuthology --verbose --name yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/orig.config.yaml 2024-05-21T17:08:09.774 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-21T17:08:09.856 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922 branch: wip-yuri3-testing-2024-05-20-1050 description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} email: yweinste@redhat.com first_in_suite: false job_id: '7716922' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri3-testing-2024-05-20-1050 ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ mon_bind_msgr2: false sha1: 99470715356be3e23220ec4a5eacc2bc4dfde33f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 99470715356be3e23220ec4a5eacc2bc4dfde33f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2024-05-20-1050 sha1: 99470715356be3e23220ec4a5eacc2bc4dfde33f owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9006 sha1: 99470715356be3e23220ec4a5eacc2bc4dfde33f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2024-05-20-1050 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_99470715356be3e23220ec4a5eacc2bc4dfde33f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 99470715356be3e23220ec4a5eacc2bc4dfde33f targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEDkmWKUi7K3N7gP72K0zGzCXHsXvDsIwLfpqskkN5nvG8JMm5kR2o89H+tQf+W9Jv2nlWfgMOzUzi0mO843moY= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIe6uKHa3bwWj2QHbMHGqluB9B9BIwqLpzyQ5jtvs+rxt7EDj/vJt75l0VWO57i4nBVWrd6K5TkeVxQkmdWWK3U= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 - exec: client.0: - ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a timestamp: 2024-05-20_20:45:46 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-21T17:08:09.857 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_99470715356be3e23220ec4a5eacc2bc4dfde33f/qa; will attempt to use it 2024-05-21T17:08:09.857 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_99470715356be3e23220ec4a5eacc2bc4dfde33f/qa/tasks 2024-05-21T17:08:09.858 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-21T17:08:09.859 INFO:teuthology.task.internal:Checking packages... 2024-05-21T17:08:09.881 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '99470715356be3e23220ec4a5eacc2bc4dfde33f' 2024-05-21T17:08:09.882 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-21T17:08:09.882 INFO:teuthology.packaging:ref: None 2024-05-21T17:08:09.882 INFO:teuthology.packaging:tag: None 2024-05-21T17:08:09.882 INFO:teuthology.packaging:branch: wip-yuri3-testing-2024-05-20-1050 2024-05-21T17:08:09.882 INFO:teuthology.packaging:sha1: 99470715356be3e23220ec4a5eacc2bc4dfde33f 2024-05-21T17:08:09.882 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri3-testing-2024-05-20-1050 2024-05-21T17:08:10.025 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3827-g99470715-1jammy 2024-05-21T17:08:10.027 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-21T17:08:10.034 INFO:teuthology.task.internal:no buildpackages task found 2024-05-21T17:08:10.034 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-21T17:08:10.068 INFO:teuthology.task.internal:Saving configuration 2024-05-21T17:08:10.080 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-21T17:08:10.127 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-21T17:08:10.152 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-21 17:02:19.572061', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIRE6jE22cl3oX0t4/8i4cjbgMVVLZHNiK+S77kQkdYo'} 2024-05-21T17:08:10.175 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-21 17:02:19.571151', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoO/UhffgrES0OdkmiGC90fWnJvQbSs1hwuik7B80IQ'} 2024-05-21T17:08:10.175 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-21T17:08:10.181 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-05-21T17:08:10.181 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-05-21T17:08:10.181 INFO:teuthology.run_tasks:Running task console_log... 2024-05-21T17:08:10.274 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc9acd5fbe0>, signals=[15]) 2024-05-21T17:08:10.275 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-21T17:08:10.281 INFO:teuthology.task.internal:Opening connections... 2024-05-21T17:08:10.281 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:08:10.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:08:10.366 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:08:10.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:08:10.447 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-21T17:08:10.454 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-05-21T17:08:10.459 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-05-21T17:08:10.459 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-05-21T17:08:10.507 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-21T17:08:10.507 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="Ubuntu" 2024-05-21T17:08:10.507 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="22.04" 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_CODENAME=jammy 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:ID=ubuntu 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE=debian 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-21T17:08:10.508 INFO:teuthology.orchestra.run.smithi003.stdout:UBUNTU_CODENAME=jammy 2024-05-21T17:08:10.509 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-05-21T17:08:10.533 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-05-21T17:08:10.538 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-05-21T17:08:10.539 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-05-21T17:08:10.585 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-21T17:08:10.585 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Ubuntu" 2024-05-21T17:08:10.585 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="22.04" 2024-05-21T17:08:10.585 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-21T17:08:10.585 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_CODENAME=jammy 2024-05-21T17:08:10.586 INFO:teuthology.orchestra.run.smithi203.stdout:ID=ubuntu 2024-05-21T17:08:10.586 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE=debian 2024-05-21T17:08:10.586 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-21T17:08:10.586 INFO:teuthology.orchestra.run.smithi203.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-21T17:08:10.586 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-21T17:08:10.586 INFO:teuthology.orchestra.run.smithi203.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-21T17:08:10.586 INFO:teuthology.orchestra.run.smithi203.stdout:UBUNTU_CODENAME=jammy 2024-05-21T17:08:10.586 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-05-21T17:08:10.608 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-21T17:08:10.616 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-21T17:08:10.642 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-21T17:08:10.643 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-05-21T17:08:10.645 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-05-21T17:08:10.649 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-21T17:08:10.676 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-21T17:08:10.676 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-05-21T17:08:10.693 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-05-21T17:08:10.706 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-21T17:08:10.755 INFO:teuthology.run_tasks:Running task kernel... 2024-05-21T17:08:10.778 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-21T17:08:10.778 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-21T17:08:10.778 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-21T17:08:10.779 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-21T17:08:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:08:10.784 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-05-21T17:08:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:08:10.785 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-05-21T17:08:10.829 INFO:teuthology.orchestra.run.smithi203.stdout:5.15.0-56-generic 2024-05-21T17:08:10.829 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.15.0-56-generic 2024-05-21T17:08:10.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-05-21T17:08:10.832 INFO:teuthology.orchestra.run.smithi003.stdout:5.15.0-56-generic 2024-05-21T17:08:10.832 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.15.0-56-generic 2024-05-21T17:08:10.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get clean 2024-05-21T17:08:10.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-05-21T17:08:10.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get update 2024-05-21T17:08:11.103 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-21T17:08:11.176 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-21T17:08:11.197 INFO:teuthology.orchestra.run.smithi003.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-21T17:08:11.205 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-21T17:08:11.269 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-21T17:08:11.276 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-21T17:08:11.287 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-21T17:08:11.306 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,470 kB] 2024-05-21T17:08:11.425 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [470 kB] 2024-05-21T17:08:11.434 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [253 kB] 2024-05-21T17:08:11.439 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-21T17:08:11.440 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-21T17:08:11.440 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,683 kB] 2024-05-21T17:08:11.441 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,871 kB] 2024-05-21T17:08:11.470 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [317 kB] 2024-05-21T17:08:11.474 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-21T17:08:11.474 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [853 kB] 2024-05-21T17:08:11.484 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [603 kB] 2024-05-21T17:08:11.489 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [164 kB] 2024-05-21T17:08:11.491 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-21T17:08:11.491 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-21T17:08:11.492 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-21T17:08:11.492 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-21T17:08:11.533 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-21T17:08:11.554 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [632 kB] 2024-05-21T17:08:11.561 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [312 kB] 2024-05-21T17:08:11.564 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-21T17:08:11.565 INFO:teuthology.orchestra.run.smithi003.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-21T17:08:11.565 INFO:teuthology.orchestra.run.smithi003.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,933 kB] 2024-05-21T17:08:11.586 INFO:teuthology.orchestra.run.smithi003.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [328 kB] 2024-05-21T17:08:11.589 INFO:teuthology.orchestra.run.smithi003.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-21T17:08:11.589 INFO:teuthology.orchestra.run.smithi003.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,075 kB] 2024-05-21T17:08:11.600 INFO:teuthology.orchestra.run.smithi003.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [702 kB] 2024-05-21T17:08:11.648 INFO:teuthology.orchestra.run.smithi003.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [246 kB] 2024-05-21T17:08:11.649 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-21T17:08:11.650 INFO:teuthology.orchestra.run.smithi003.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-21T17:08:11.650 INFO:teuthology.orchestra.run.smithi003.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-21T17:08:11.651 INFO:teuthology.orchestra.run.smithi003.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-21T17:08:11.651 INFO:teuthology.orchestra.run.smithi003.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-21T17:08:11.652 INFO:teuthology.orchestra.run.smithi003.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-21T17:08:11.652 INFO:teuthology.orchestra.run.smithi003.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-21T17:08:11.653 INFO:teuthology.orchestra.run.smithi003.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-21T17:08:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-21T17:08:11.670 INFO:teuthology.orchestra.run.smithi003.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-21T17:08:11.691 INFO:teuthology.orchestra.run.smithi003.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-21T17:08:11.691 INFO:teuthology.orchestra.run.smithi003.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-21T17:08:11.692 INFO:teuthology.orchestra.run.smithi003.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-21T17:08:11.692 INFO:teuthology.orchestra.run.smithi003.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-21T17:08:11.744 INFO:teuthology.orchestra.run.smithi003.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [470 kB] 2024-05-21T17:08:11.787 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [632 kB] 2024-05-21T17:08:11.989 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,683 kB] 2024-05-21T17:08:12.036 INFO:teuthology.orchestra.run.smithi003.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,470 kB] 2024-05-21T17:08:12.146 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [312 kB] 2024-05-21T17:08:12.161 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-21T17:08:12.162 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-21T17:08:12.163 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,933 kB] 2024-05-21T17:08:12.213 INFO:teuthology.orchestra.run.smithi003.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [253 kB] 2024-05-21T17:08:12.226 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [328 kB] 2024-05-21T17:08:12.226 INFO:teuthology.orchestra.run.smithi003.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-21T17:08:12.226 INFO:teuthology.orchestra.run.smithi003.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,871 kB] 2024-05-21T17:08:12.233 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-21T17:08:12.310 INFO:teuthology.orchestra.run.smithi003.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-21T17:08:12.310 INFO:teuthology.orchestra.run.smithi003.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [317 kB] 2024-05-21T17:08:12.319 INFO:teuthology.orchestra.run.smithi003.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-21T17:08:12.320 INFO:teuthology.orchestra.run.smithi003.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [853 kB] 2024-05-21T17:08:12.341 INFO:teuthology.orchestra.run.smithi003.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [603 kB] 2024-05-21T17:08:12.398 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,075 kB] 2024-05-21T17:08:12.517 INFO:teuthology.orchestra.run.smithi003.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [164 kB] 2024-05-21T17:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-21T17:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-21T17:08:12.944 INFO:teuthology.orchestra.run.smithi003.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-21T17:08:12.944 INFO:teuthology.orchestra.run.smithi003.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-21T17:08:12.947 INFO:teuthology.orchestra.run.smithi003.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-21T17:08:13.033 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [702 kB] 2024-05-21T17:08:13.074 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [246 kB] 2024-05-21T17:08:13.090 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-21T17:08:13.091 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-21T17:08:13.094 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-21T17:08:13.094 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-21T17:08:13.095 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-21T17:08:13.095 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-21T17:08:13.116 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-21T17:08:13.119 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-21T17:08:13.284 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-21T17:08:13.366 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-21T17:08:13.468 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-21T17:08:13.502 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-21T17:08:13.553 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-21T17:08:17.631 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 13.7 MB in 2s (6,685 kB/s) 2024-05-21T17:08:17.782 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 13.7 MB in 3s (5,420 kB/s) 2024-05-21T17:08:19.155 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-05-21T17:08:19.167 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-05-21T17:08:19.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-21T17:08:19.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-21T17:08:19.235 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-05-21T17:08:19.242 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-05-21T17:08:19.440 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2024-05-21T17:08:19.441 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2024-05-21T17:08:19.441 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-05-21T17:08:19.442 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-05-21T17:08:19.609 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-05-21T17:08:19.609 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-21T17:08:19.609 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-21T17:08:19.610 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-21T17:08:19.610 INFO:teuthology.orchestra.run.smithi203.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-21T17:08:19.610 INFO:teuthology.orchestra.run.smithi203.stdout: python2.7-minimal 2024-05-21T17:08:19.611 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-21T17:08:19.611 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-05-21T17:08:19.611 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-5.15.0-107 linux-headers-5.15.0-107-generic 2024-05-21T17:08:19.611 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-generic linux-image-5.15.0-107-generic 2024-05-21T17:08:19.611 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-5.15.0-107-generic linux-modules-extra-5.15.0-107-generic 2024-05-21T17:08:19.613 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-05-21T17:08:19.613 INFO:teuthology.orchestra.run.smithi203.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-21T17:08:19.613 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-05-21T17:08:19.613 INFO:teuthology.orchestra.run.smithi203.stdout: thermald 2024-05-21T17:08:19.624 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages were automatically installed and are no longer required: 2024-05-21T17:08:19.624 INFO:teuthology.orchestra.run.smithi003.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-21T17:08:19.624 INFO:teuthology.orchestra.run.smithi003.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-21T17:08:19.625 INFO:teuthology.orchestra.run.smithi003.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-21T17:08:19.625 INFO:teuthology.orchestra.run.smithi003.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-21T17:08:19.625 INFO:teuthology.orchestra.run.smithi003.stdout: python2.7-minimal 2024-05-21T17:08:19.625 INFO:teuthology.orchestra.run.smithi003.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-21T17:08:19.626 INFO:teuthology.orchestra.run.smithi003.stdout:The following additional packages will be installed: 2024-05-21T17:08:19.626 INFO:teuthology.orchestra.run.smithi003.stdout: linux-generic linux-headers-5.15.0-107 linux-headers-5.15.0-107-generic 2024-05-21T17:08:19.627 INFO:teuthology.orchestra.run.smithi003.stdout: linux-headers-generic linux-image-5.15.0-107-generic 2024-05-21T17:08:19.627 INFO:teuthology.orchestra.run.smithi003.stdout: linux-modules-5.15.0-107-generic linux-modules-extra-5.15.0-107-generic 2024-05-21T17:08:19.628 INFO:teuthology.orchestra.run.smithi003.stdout:Suggested packages: 2024-05-21T17:08:19.628 INFO:teuthology.orchestra.run.smithi003.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-21T17:08:19.628 INFO:teuthology.orchestra.run.smithi003.stdout:Recommended packages: 2024-05-21T17:08:19.629 INFO:teuthology.orchestra.run.smithi003.stdout: thermald 2024-05-21T17:08:19.670 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-05-21T17:08:19.671 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-5.15.0-107 linux-headers-5.15.0-107-generic 2024-05-21T17:08:19.671 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.15.0-107-generic linux-modules-5.15.0-107-generic 2024-05-21T17:08:19.671 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-extra-5.15.0-107-generic 2024-05-21T17:08:19.671 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages will be upgraded: 2024-05-21T17:08:19.672 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-21T17:08:19.680 INFO:teuthology.orchestra.run.smithi003.stdout:The following NEW packages will be installed: 2024-05-21T17:08:19.680 INFO:teuthology.orchestra.run.smithi003.stdout: linux-headers-5.15.0-107 linux-headers-5.15.0-107-generic 2024-05-21T17:08:19.680 INFO:teuthology.orchestra.run.smithi003.stdout: linux-image-5.15.0-107-generic linux-modules-5.15.0-107-generic 2024-05-21T17:08:19.681 INFO:teuthology.orchestra.run.smithi003.stdout: linux-modules-extra-5.15.0-107-generic 2024-05-21T17:08:19.682 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages will be upgraded: 2024-05-21T17:08:19.683 INFO:teuthology.orchestra.run.smithi003.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-21T17:08:19.741 INFO:teuthology.orchestra.run.smithi003.stdout:3 upgraded, 5 newly installed, 0 to remove and 320 not upgraded. 2024-05-21T17:08:19.741 INFO:teuthology.orchestra.run.smithi003.stdout:Need to get 113 MB of archives. 2024-05-21T17:08:19.741 INFO:teuthology.orchestra.run.smithi003.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-21T17:08:19.741 INFO:teuthology.orchestra.run.smithi003.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-107-generic amd64 5.15.0-107.117 [22.7 MB] 2024-05-21T17:08:19.851 INFO:teuthology.orchestra.run.smithi203.stdout:3 upgraded, 5 newly installed, 0 to remove and 320 not upgraded. 2024-05-21T17:08:19.851 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 113 MB of archives. 2024-05-21T17:08:19.851 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-21T17:08:19.851 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-107-generic amd64 5.15.0-107.117 [22.7 MB] 2024-05-21T17:08:20.253 INFO:teuthology.orchestra.run.smithi003.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-107-generic amd64 5.15.0-107.117 [11.5 MB] 2024-05-21T17:08:20.370 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-107-generic amd64 5.15.0-107.117 [63.9 MB] 2024-05-21T17:08:21.322 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.107.107 [1,694 B] 2024-05-21T17:08:21.323 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.107.107 [2,496 B] 2024-05-21T17:08:21.323 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-107 all 5.15.0-107.117 [12.3 MB] 2024-05-21T17:08:21.382 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-107-generic amd64 5.15.0-107.117 [11.5 MB] 2024-05-21T17:08:21.474 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-107-generic amd64 5.15.0-107.117 [2,887 kB] 2024-05-21T17:08:21.509 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.107.107 [2,354 B] 2024-05-21T17:08:21.736 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-107-generic amd64 5.15.0-107.117 [63.9 MB] 2024-05-21T17:08:21.935 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 113 MB in 2s (62.3 MB/s) 2024-05-21T17:08:22.129 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-modules-5.15.0-107-generic. 2024-05-21T17:08:24.005 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.107.107 [1,694 B] 2024-05-21T17:08:24.006 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.107.107 [2,496 B] 2024-05-21T17:08:24.006 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-107 all 5.15.0-107.117 [12.3 MB] 2024-05-21T17:08:24.127 INFO:teuthology.orchestra.run.smithi003.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-21T17:08:24.131 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../0-linux-modules-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-21T17:08:24.228 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-modules-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:24.552 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-107-generic amd64 5.15.0-107.117 [2,887 kB] 2024-05-21T17:08:24.650 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.107.107 [2,354 B] 2024-05-21T17:08:25.047 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 113 MB in 5s (22.8 MB/s) 2024-05-21T17:08:25.232 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-5.15.0-107-generic. 2024-05-21T17:08:26.470 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-image-5.15.0-107-generic. 2024-05-21T17:08:26.492 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../1-linux-image-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-21T17:08:26.780 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:27.260 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-21T17:08:27.264 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../0-linux-modules-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-21T17:08:27.328 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:27.365 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-107-generic. 2024-05-21T17:08:27.386 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-21T17:08:27.423 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-modules-extra-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:29.488 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-image-5.15.0-107-generic. 2024-05-21T17:08:29.509 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../1-linux-image-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-21T17:08:29.613 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:30.016 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-107-generic. 2024-05-21T17:08:30.037 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-21T17:08:30.074 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-extra-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:34.046 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../3-linux-generic_5.15.0.107.107_amd64.deb ... 2024-05-21T17:08:34.172 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-21T17:08:34.480 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.107.107_amd64.deb ... 2024-05-21T17:08:34.600 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-image-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-21T17:08:34.937 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-headers-5.15.0-107. 2024-05-21T17:08:34.959 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../5-linux-headers-5.15.0-107_5.15.0-107.117_all.deb ... 2024-05-21T17:08:35.003 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-headers-5.15.0-107 (5.15.0-107.117) ... 2024-05-21T17:08:36.838 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../3-linux-generic_5.15.0.107.107_amd64.deb ... 2024-05-21T17:08:36.965 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-21T17:08:37.248 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.107.107_amd64.deb ... 2024-05-21T17:08:37.377 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-21T17:08:37.664 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.15.0-107. 2024-05-21T17:08:37.686 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../5-linux-headers-5.15.0-107_5.15.0-107.117_all.deb ... 2024-05-21T17:08:37.730 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.15.0-107 (5.15.0-107.117) ... 2024-05-21T17:08:40.688 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-headers-5.15.0-107-generic. 2024-05-21T17:08:40.726 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../6-linux-headers-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-21T17:08:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-headers-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:43.496 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.15.0-107-generic. 2024-05-21T17:08:43.533 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../6-linux-headers-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-21T17:08:43.562 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:44.190 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.107.107_amd64.deb ... 2024-05-21T17:08:44.340 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-headers-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-21T17:08:44.690 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-headers-5.15.0-107 (5.15.0-107.117) ... 2024-05-21T17:08:44.836 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-headers-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:44.962 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-headers-generic (5.15.0.107.107) ... 2024-05-21T17:08:45.091 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:46.261 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.107.107_amd64.deb ... 2024-05-21T17:08:46.388 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-21T17:08:46.448 INFO:teuthology.orchestra.run.smithi003.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-21T17:08:46.448 INFO:teuthology.orchestra.run.smithi003.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-21T17:08:46.448 INFO:teuthology.orchestra.run.smithi003.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-107-generic 2024-05-21T17:08:46.448 INFO:teuthology.orchestra.run.smithi003.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-107-generic 2024-05-21T17:08:46.618 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-modules-extra-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:47.506 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.15.0-107 (5.15.0-107.117) ... 2024-05-21T17:08:47.611 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:47.712 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-generic (5.15.0.107.107) ... 2024-05-21T17:08:47.841 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:49.035 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-modules-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:49.165 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-21T17:08:49.165 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-21T17:08:49.165 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-107-generic 2024-05-21T17:08:49.165 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-107-generic 2024-05-21T17:08:49.320 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-extra-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:50.445 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-image-generic (5.15.0.107.107) ... 2024-05-21T17:08:50.571 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-generic (5.15.0.107.107) ... 2024-05-21T17:08:50.731 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:50.737 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:50.793 INFO:teuthology.orchestra.run.smithi003.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-21T17:08:50.793 INFO:teuthology.orchestra.run.smithi003.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-107-generic 2024-05-21T17:08:52.155 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-generic (5.15.0.107.107) ... 2024-05-21T17:08:52.281 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-generic (5.15.0.107.107) ... 2024-05-21T17:08:52.583 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-21T17:08:53.239 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-21T17:08:53.239 INFO:teuthology.orchestra.run.smithi203.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-107-generic 2024-05-21T17:09:05.581 INFO:teuthology.orchestra.run.smithi003.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-21T17:09:05.581 INFO:teuthology.orchestra.run.smithi003.stdout:Sourcing file `/etc/default/grub' 2024-05-21T17:09:05.597 INFO:teuthology.orchestra.run.smithi003.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-21T17:09:05.620 INFO:teuthology.orchestra.run.smithi003.stdout:Generating grub configuration file ... 2024-05-21T17:09:05.935 INFO:teuthology.orchestra.run.smithi003.stdout:Found linux image: /boot/vmlinuz-5.15.0-107-generic 2024-05-21T17:09:05.954 INFO:teuthology.orchestra.run.smithi003.stdout:Found initrd image: /boot/initrd.img-5.15.0-107-generic 2024-05-21T17:09:06.232 INFO:teuthology.orchestra.run.smithi003.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-21T17:09:06.237 INFO:teuthology.orchestra.run.smithi003.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-21T17:09:06.302 INFO:teuthology.orchestra.run.smithi003.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-21T17:09:06.308 INFO:teuthology.orchestra.run.smithi003.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-21T17:09:06.489 INFO:teuthology.orchestra.run.smithi003.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-21T17:09:06.489 INFO:teuthology.orchestra.run.smithi003.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-21T17:09:06.489 INFO:teuthology.orchestra.run.smithi003.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-21T17:09:06.534 INFO:teuthology.orchestra.run.smithi003.stdout:done 2024-05-21T17:09:07.012 DEBUG:teuthology.orchestra.run.smithi003:> dpkg -s linux-image-generic 2024-05-21T17:09:07.035 INFO:teuthology.orchestra.run.smithi003.stdout:Package: linux-image-generic 2024-05-21T17:09:07.035 INFO:teuthology.orchestra.run.smithi003.stdout:Status: install ok installed 2024-05-21T17:09:07.035 INFO:teuthology.orchestra.run.smithi003.stdout:Priority: optional 2024-05-21T17:09:07.035 INFO:teuthology.orchestra.run.smithi003.stdout:Section: kernel 2024-05-21T17:09:07.035 INFO:teuthology.orchestra.run.smithi003.stdout:Installed-Size: 21 2024-05-21T17:09:07.035 INFO:teuthology.orchestra.run.smithi003.stdout:Maintainer: Ubuntu Kernel Team 2024-05-21T17:09:07.035 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: amd64 2024-05-21T17:09:07.036 INFO:teuthology.orchestra.run.smithi003.stdout:Source: linux-meta 2024-05-21T17:09:07.036 INFO:teuthology.orchestra.run.smithi003.stdout:Version: 5.15.0.107.107 2024-05-21T17:09:07.036 INFO:teuthology.orchestra.run.smithi003.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-107), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-21T17:09:07.036 INFO:teuthology.orchestra.run.smithi003.stdout:Depends: linux-image-5.15.0-107-generic, linux-modules-extra-5.15.0-107-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-21T17:09:07.036 INFO:teuthology.orchestra.run.smithi003.stdout:Recommends: thermald 2024-05-21T17:09:07.036 INFO:teuthology.orchestra.run.smithi003.stdout:Description: Generic Linux kernel image 2024-05-21T17:09:07.036 INFO:teuthology.orchestra.run.smithi003.stdout: This package will always depend on the latest generic kernel image 2024-05-21T17:09:07.036 INFO:teuthology.orchestra.run.smithi003.stdout: available. 2024-05-21T17:09:07.036 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-107-generic 2024-05-21T17:09:07.036 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-21T17:09:07.037 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-21T17:09:07.037 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-21T17:09:07.037 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-05-21T17:09:07.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get clean 2024-05-21T17:09:07.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get update 2024-05-21T17:09:07.488 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-21T17:09:07.594 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-21T17:09:07.674 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-21T17:09:07.711 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-21T17:09:07.711 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub' 2024-05-21T17:09:07.724 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-21T17:09:07.742 INFO:teuthology.orchestra.run.smithi203.stdout:Generating grub configuration file ... 2024-05-21T17:09:07.756 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-21T17:09:07.963 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.15.0-107-generic 2024-05-21T17:09:07.982 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.15.0-107-generic 2024-05-21T17:09:08.284 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-21T17:09:08.290 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-21T17:09:08.360 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-21T17:09:08.365 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-21T17:09:08.512 INFO:teuthology.orchestra.run.smithi203.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-21T17:09:08.512 INFO:teuthology.orchestra.run.smithi203.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-21T17:09:08.512 INFO:teuthology.orchestra.run.smithi203.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-21T17:09:08.542 INFO:teuthology.orchestra.run.smithi203.stdout:done 2024-05-21T17:09:08.908 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-05-21T17:09:08.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-21T17:09:08.986 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-05-21T17:09:08.999 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-05-21T17:09:09.058 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 21 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.15.0.107.107 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-107), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-21T17:09:09.059 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.15.0-107-generic, linux-modules-extra-5.15.0-107-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-21T17:09:09.060 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-05-21T17:09:09.060 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-05-21T17:09:09.060 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-05-21T17:09:09.060 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-05-21T17:09:09.060 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-107-generic 2024-05-21T17:09:09.060 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-21T17:09:09.060 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-21T17:09:09.060 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2024-05-21T17:09:09.060 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-05-21T17:09:09.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-05-21T17:09:09.187 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2024-05-21T17:09:09.187 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2024-05-21T17:09:09.240 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-05-21T17:09:09.375 INFO:teuthology.orchestra.run.smithi003.stdout:linux-image-generic is already the newest version (5.15.0.107.107). 2024-05-21T17:09:09.376 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages were automatically installed and are no longer required: 2024-05-21T17:09:09.376 INFO:teuthology.orchestra.run.smithi003.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-21T17:09:09.376 INFO:teuthology.orchestra.run.smithi003.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-21T17:09:09.376 INFO:teuthology.orchestra.run.smithi003.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-21T17:09:09.376 INFO:teuthology.orchestra.run.smithi003.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-21T17:09:09.377 INFO:teuthology.orchestra.run.smithi003.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-21T17:09:09.377 INFO:teuthology.orchestra.run.smithi003.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-21T17:09:09.418 INFO:teuthology.orchestra.run.smithi003.stdout:0 upgraded, 0 newly installed, 0 to remove and 320 not upgraded. 2024-05-21T17:09:09.420 DEBUG:teuthology.orchestra.run.smithi003:> dpkg -s linux-image-generic 2024-05-21T17:09:09.436 INFO:teuthology.orchestra.run.smithi003.stdout:Package: linux-image-generic 2024-05-21T17:09:09.436 INFO:teuthology.orchestra.run.smithi003.stdout:Status: install ok installed 2024-05-21T17:09:09.436 INFO:teuthology.orchestra.run.smithi003.stdout:Priority: optional 2024-05-21T17:09:09.436 INFO:teuthology.orchestra.run.smithi003.stdout:Section: kernel 2024-05-21T17:09:09.436 INFO:teuthology.orchestra.run.smithi003.stdout:Installed-Size: 21 2024-05-21T17:09:09.436 INFO:teuthology.orchestra.run.smithi003.stdout:Maintainer: Ubuntu Kernel Team 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: amd64 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout:Source: linux-meta 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout:Version: 5.15.0.107.107 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-107), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout:Depends: linux-image-5.15.0-107-generic, linux-modules-extra-5.15.0-107-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout:Recommends: thermald 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout:Description: Generic Linux kernel image 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout: This package will always depend on the latest generic kernel image 2024-05-21T17:09:09.437 INFO:teuthology.orchestra.run.smithi003.stdout: available. 2024-05-21T17:09:09.437 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-05-21T17:09:09.483 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.B8vVM3VQiN 2024-05-21T17:09:09.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub/grub.cfg /tmp/tmp.B8vVM3VQiN 2024-05-21T17:09:09.488 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-21T17:09:09.488 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-21T17:09:09.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.B8vVM3VQiN 2024-05-21T17:09:09.561 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-21T17:09:09.643 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-21T17:09:09.651 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.B8vVM3VQiN is 10KB 2024-05-21T17:09:09.664 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.B8vVM3VQiN 2024-05-21T17:09:09.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-21T17:09:09.723 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:09:09.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-21T17:09:09.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-21T17:09:09.813 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-107-generic 2024-05-21T17:09:09.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo update-grub 2024-05-21T17:09:10.891 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-05-21T17:09:10.909 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-21T17:09:10.970 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-05-21T17:09:11.115 INFO:teuthology.orchestra.run.smithi003.stderr:Sourcing file `/etc/default/grub' 2024-05-21T17:09:11.116 INFO:teuthology.orchestra.run.smithi003.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-21T17:09:11.119 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-05-21T17:09:11.173 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-05-21T17:09:11.173 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-05-21T17:09:11.341 INFO:teuthology.orchestra.run.smithi003.stderr:Found linux image: /boot/vmlinuz-5.15.0-107-generic 2024-05-21T17:09:11.347 INFO:teuthology.orchestra.run.smithi003.stderr:Found initrd image: /boot/initrd.img-5.15.0-107-generic 2024-05-21T17:09:11.354 INFO:teuthology.orchestra.run.smithi203.stdout:linux-image-generic is already the newest version (5.15.0.107.107). 2024-05-21T17:09:11.354 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-05-21T17:09:11.354 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-21T17:09:11.354 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-21T17:09:11.354 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-21T17:09:11.355 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-21T17:09:11.355 INFO:teuthology.orchestra.run.smithi203.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-21T17:09:11.355 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-21T17:09:11.398 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 0 newly installed, 0 to remove and 320 not upgraded. 2024-05-21T17:09:11.400 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-05-21T17:09:11.416 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-05-21T17:09:11.416 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-05-21T17:09:11.416 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-05-21T17:09:11.416 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-05-21T17:09:11.416 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 21 2024-05-21T17:09:11.416 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.15.0.107.107 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-107), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.15.0-107-generic, linux-modules-extra-5.15.0-107-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-05-21T17:09:11.417 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-05-21T17:09:11.417 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-05-21T17:09:11.464 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.aZLMFcYB5L 2024-05-21T17:09:11.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub/grub.cfg /tmp/tmp.aZLMFcYB5L 2024-05-21T17:09:11.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.aZLMFcYB5L 2024-05-21T17:09:11.593 INFO:teuthology.orchestra.run.smithi003.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-21T17:09:11.600 INFO:teuthology.orchestra.run.smithi003.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-21T17:09:11.642 INFO:teuthology.orchestra.run.smithi003.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-21T17:09:11.648 INFO:teuthology.orchestra.run.smithi003.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-21T17:09:11.655 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.aZLMFcYB5L is 10KB 2024-05-21T17:09:11.668 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.aZLMFcYB5L 2024-05-21T17:09:11.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-21T17:09:11.727 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:09:11.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-21T17:09:11.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-21T17:09:11.791 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-107-generic 2024-05-21T17:09:11.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo update-grub 2024-05-21T17:09:11.819 INFO:teuthology.orchestra.run.smithi003.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-21T17:09:11.819 INFO:teuthology.orchestra.run.smithi003.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-21T17:09:11.820 INFO:teuthology.orchestra.run.smithi003.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-21T17:09:11.837 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-05-21T17:09:11.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-05-21T17:09:12.998 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub' 2024-05-21T17:09:12.999 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-21T17:09:13.002 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-05-21T17:09:13.173 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.15.0-107-generic 2024-05-21T17:09:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.15.0-107-generic 2024-05-21T17:09:13.410 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-21T17:09:13.417 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-21T17:09:13.457 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-21T17:09:13.462 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-21T17:09:13.618 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-21T17:09:13.619 INFO:teuthology.orchestra.run.smithi203.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-21T17:09:13.619 INFO:teuthology.orchestra.run.smithi203.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-21T17:09:13.640 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-05-21T17:09:13.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-05-21T17:09:41.870 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-21T17:09:41.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-05-21T17:09:41.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:09:43.654 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2024-05-21T17:09:43.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-05-21T17:09:43.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:10:00.223 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-05-21T17:10:02.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-05-21T17:10:09.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-05-21T17:10:09.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:10:11.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-05-21T17:10:11.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:10:12.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-05-21T17:10:14.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-05-21T17:10:24.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-05-21T17:10:24.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:10:26.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-05-21T17:10:26.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:10:56.095 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-05-21T17:10:56.882 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-05-21T17:10:56.882 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-107-generic"... 2024-05-21T17:10:56.883 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-05-21T17:10:56.928 INFO:teuthology.orchestra.run.smithi003.stdout:5.15.0-107-generic 2024-05-21T17:10:56.929 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-107-generic vs 5.15.0-107-generic 2024-05-21T17:10:56.929 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-21T17:10:56.929 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-21T17:10:57.929 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-21T17:10:57.930 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-21T17:10:58.030 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-05-21T17:10:58.050 DEBUG:teuthology.parallel:result is None 2024-05-21T17:11:26.085 DEBUG:teuthology.orchestra.remote:timed out 2024-05-21T17:11:41.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-05-21T17:11:41.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:11:41.403 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-05-21T17:11:41.959 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-05-21T17:11:41.959 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "5.15.0-107-generic"... 2024-05-21T17:11:41.959 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-05-21T17:11:42.005 INFO:teuthology.orchestra.run.smithi203.stdout:5.15.0-107-generic 2024-05-21T17:11:42.006 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-107-generic vs 5.15.0-107-generic 2024-05-21T17:11:42.006 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-21T17:11:42.006 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-21T17:11:43.007 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-05-21T17:11:43.007 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-21T17:11:43.386 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-05-21T17:11:43.419 DEBUG:teuthology.parallel:result is None 2024-05-21T17:11:43.420 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-21T17:11:43.427 INFO:teuthology.task.internal:Creating test directory... 2024-05-21T17:11:43.428 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-21T17:11:43.430 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-21T17:11:43.436 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-21T17:11:43.469 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-21T17:11:43.510 INFO:teuthology.task.internal:Creating archive directory... 2024-05-21T17:11:43.511 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-21T17:11:43.513 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-21T17:11:43.539 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-21T17:11:43.552 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-21T17:11:43.553 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-21T17:11:43.566 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-21T17:11:43.580 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-21T17:11:43.588 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-21T17:11:43.592 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-21T17:11:43.602 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-21T17:11:43.603 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-21T17:11:43.609 INFO:teuthology.task.internal:Configuring sudo... 2024-05-21T17:11:43.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-21T17:11:43.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-21T17:11:43.661 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-21T17:11:43.670 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-21T17:11:43.670 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-21T17:11:43.694 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-21T17:11:43.710 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-21T17:11:43.745 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-21T17:11:43.793 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:11:43.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-21T17:11:43.859 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-21T17:11:43.865 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-21T17:11:43.915 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:11:43.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-21T17:11:43.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-05-21T17:11:43.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-05-21T17:11:44.079 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-21T17:11:44.086 INFO:teuthology.task.internal:Starting timer... 2024-05-21T17:11:44.086 INFO:teuthology.run_tasks:Running task pcp... 2024-05-21T17:11:44.123 INFO:teuthology.run_tasks:Running task selinux... 2024-05-21T17:11:44.171 DEBUG:teuthology.task.selinux:Excluding smithi003: OS 'ubuntu' does not support SELinux 2024-05-21T17:11:44.171 DEBUG:teuthology.task.selinux:Excluding smithi203: OS 'ubuntu' does not support SELinux 2024-05-21T17:11:44.172 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-21T17:11:44.172 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-05-21T17:11:44.172 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-21T17:11:44.172 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-21T17:11:44.266 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-21T17:11:44.277 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-21T17:11:44.278 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-21T17:18:50.418 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-05-21T17:18:50.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-05-21T17:18:50.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:18:50.504 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-05-21T17:18:50.583 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-05-21T17:18:50.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-05-21T17:18:50.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T17:18:50.663 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-05-21T17:18:50.742 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-05-21T17:18:50.743 INFO:teuthology.run_tasks:Running task clock... 2024-05-21T17:18:50.758 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-21T17:18:50.758 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-21T17:18:50.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-21T17:18:50.761 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-21T17:18:50.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-21T17:18:50.791 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-21T17:18:50.791 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Command line: ntpd -gq 2024-05-21T17:18:50.791 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: ---------------------------------------------------- 2024-05-21T17:18:50.791 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: ntp-4 is maintained by Network Time Foundation, 2024-05-21T17:18:50.791 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-21T17:18:50.791 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: corporation. Support and training for ntp-4 are 2024-05-21T17:18:50.791 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: available at https://www.nwtime.org/support 2024-05-21T17:18:50.791 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: ---------------------------------------------------- 2024-05-21T17:18:50.792 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: proto: precision = 0.066 usec (-24) 2024-05-21T17:18:50.792 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: basedate set to 2022-02-04 2024-05-21T17:18:50.792 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: gps base set to 2022-02-06 (week 2196) 2024-05-21T17:18:50.792 INFO:teuthology.orchestra.run.smithi003.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-21T17:18:50.793 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-21T17:18:50.793 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: restrict ::: KOD does nothing without LIMITED. 2024-05-21T17:18:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-21T17:18:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Listen and drop on 0 v6wildcard [::]:123 2024-05-21T17:18:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-21T17:18:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Listen normally on 2 lo 127.0.0.1:123 2024-05-21T17:18:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Listen normally on 3 ens1f0 172.21.15.3:123 2024-05-21T17:18:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Listen normally on 4 lo [::1]:123 2024-05-21T17:18:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15dc%4]:123 2024-05-21T17:18:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:50 ntpd[17470]: Listening on routing socket on fd #22 for interface updates 2024-05-21T17:18:50.809 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-21T17:18:50.809 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Command line: ntpd -gq 2024-05-21T17:18:50.809 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: ---------------------------------------------------- 2024-05-21T17:18:50.809 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: ntp-4 is maintained by Network Time Foundation, 2024-05-21T17:18:50.809 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-21T17:18:50.809 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: corporation. Support and training for ntp-4 are 2024-05-21T17:18:50.809 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: available at https://www.nwtime.org/support 2024-05-21T17:18:50.809 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: ---------------------------------------------------- 2024-05-21T17:18:50.810 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: proto: precision = 0.045 usec (-24) 2024-05-21T17:18:50.810 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: basedate set to 2022-02-04 2024-05-21T17:18:50.810 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: gps base set to 2022-02-06 (week 2196) 2024-05-21T17:18:50.810 INFO:teuthology.orchestra.run.smithi203.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-21T17:18:50.811 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-21T17:18:50.811 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: restrict ::: KOD does nothing without LIMITED. 2024-05-21T17:18:50.811 INFO:teuthology.orchestra.run.smithi203.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-21T17:18:50.812 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Listen and drop on 0 v6wildcard [::]:123 2024-05-21T17:18:50.812 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-21T17:18:50.812 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Listen normally on 2 lo 127.0.0.1:123 2024-05-21T17:18:50.812 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Listen normally on 3 enp3s0f1 172.21.15.203:123 2024-05-21T17:18:50.812 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Listen normally on 4 lo [::1]:123 2024-05-21T17:18:50.812 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4a23%5]:123 2024-05-21T17:18:50.813 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:50 ntpd[17585]: Listening on routing socket on fd #22 for interface updates 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:53 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:53.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:53 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:53.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:53 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:53.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:53 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:53.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:53 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:53.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:53 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:53.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:53 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:54.791 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:54 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:54.791 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:54 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:54.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:54 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:54.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:54 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:54.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:54 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:54.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:54 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:54.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:54 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:54.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:54 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:54.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:54 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:54.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:54 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:55 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:55.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:55.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:55.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:55.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:55.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:55.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:55 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:56.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:56 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:56.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:56 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:56.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:56 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:56.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:56 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:56.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:56 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:56.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:56 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:57.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:57.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:57.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:57.794 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:57 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:57.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:57.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:57.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:57.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:57.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:57.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:57 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:58.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:58 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:58.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:58 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:58.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:58 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:58.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:58 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:58.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:58 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:58.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:58 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:59.792 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:59.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:59.793 INFO:teuthology.orchestra.run.smithi003.stdout:21 May 17:18:59 ntpd[17470]: ntpd: time slew -0.001405 s 2024-05-21T17:18:59.793 INFO:teuthology.orchestra.run.smithi003.stdout:ntpd: time slew -0.001405s 2024-05-21T17:18:59.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-21T17:18:59.793 INFO:teuthology.orchestra.run.smithi003.stderr:21 May 17:18:59 ntpd[17470]: can't open /var/log/ntpstats/loopstats.20240521: Permission denied 2024-05-21T17:18:59.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:59.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:59.810 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: can't open /var/log/ntpstats/rawstats.20240521: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: can't open /var/log/ntpstats/peerstats.20240521: Permission denied 2024-05-21T17:18:59.811 INFO:teuthology.orchestra.run.smithi203.stdout:21 May 17:18:59 ntpd[17585]: ntpd: time slew -0.000288 s 2024-05-21T17:18:59.812 INFO:teuthology.orchestra.run.smithi203.stdout:ntpd: time slew -0.000288s 2024-05-21T17:18:59.812 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-21T17:18:59.812 INFO:teuthology.orchestra.run.smithi203.stderr:21 May 17:18:59 ntpd[17585]: can't open /var/log/ntpstats/loopstats.20240521: Permission denied 2024-05-21T17:18:59.839 INFO:teuthology.orchestra.run.smithi003.stdout: remote refid st t when poll reach delay offset jitter 2024-05-21T17:18:59.839 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================== 2024-05-21T17:18:59.839 INFO:teuthology.orchestra.run.smithi003.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:18:59.839 INFO:teuthology.orchestra.run.smithi003.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:18:59.839 INFO:teuthology.orchestra.run.smithi003.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:18:59.840 INFO:teuthology.orchestra.run.smithi003.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:18:59.868 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-05-21T17:18:59.868 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-05-21T17:18:59.868 INFO:teuthology.orchestra.run.smithi203.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:18:59.868 INFO:teuthology.orchestra.run.smithi203.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:18:59.869 INFO:teuthology.orchestra.run.smithi203.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:18:59.869 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:18:59.869 INFO:teuthology.run_tasks:Running task install... 2024-05-21T17:18:59.877 DEBUG:teuthology.task.install:project ceph 2024-05-21T17:18:59.877 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '99470715356be3e23220ec4a5eacc2bc4dfde33f'}} 2024-05-21T17:18:59.877 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '99470715356be3e23220ec4a5eacc2bc4dfde33f'} 2024-05-21T17:18:59.877 INFO:teuthology.task.install:Using flavor: default 2024-05-21T17:18:59.883 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-21T17:18:59.883 INFO:teuthology.task.install:extra packages: [] 2024-05-21T17:18:59.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-key list | grep Ceph 2024-05-21T17:18:59.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-key list | grep Ceph 2024-05-21T17:18:59.941 INFO:teuthology.orchestra.run.smithi003.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-21T17:18:59.969 INFO:teuthology.orchestra.run.smithi003.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-21T17:18:59.969 INFO:teuthology.orchestra.run.smithi003.stdout:uid [ unknown] Ceph.com (release key) 2024-05-21T17:18:59.969 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-05-21T17:18:59.970 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=99470715356be3e23220ec4a5eacc2bc4dfde33f 2024-05-21T17:18:59.974 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-21T17:18:59.999 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-21T17:18:59.999 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph.com (release key) 2024-05-21T17:19:00.000 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-05-21T17:19:00.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=99470715356be3e23220ec4a5eacc2bc4dfde33f 2024-05-21T17:19:00.186 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default/ 2024-05-21T17:19:00.187 INFO:teuthology.task.install.deb:Package version is 19.0.0-3827-g99470715-1jammy 2024-05-21T17:19:00.217 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default/ 2024-05-21T17:19:00.217 INFO:teuthology.task.install.deb:Package version is 19.0.0-3827-g99470715-1jammy 2024-05-21T17:19:00.330 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:19:00.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-21T17:19:00.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-05-21T17:19:00.356 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:19:00.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-21T17:19:00.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get update 2024-05-21T17:19:00.562 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-21T17:19:00.563 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-21T17:19:00.567 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy InRelease 2024-05-21T17:19:00.574 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-21T17:19:00.589 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-21T17:19:00.591 INFO:teuthology.orchestra.run.smithi003.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy InRelease 2024-05-21T17:19:00.608 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-21T17:19:00.611 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-21T17:19:00.612 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-21T17:19:00.633 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-05-21T17:19:00.634 INFO:teuthology.orchestra.run.smithi003.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-21T17:19:00.655 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-05-21T17:19:00.660 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,405 B] 2024-05-21T17:19:00.661 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-21T17:19:00.663 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:9 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-21T17:19:00.682 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,405 B] 2024-05-21T17:19:00.744 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-21T17:19:00.826 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-21T17:19:00.932 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 29.2 kB in 0s (71.3 kB/s) 2024-05-21T17:19:00.969 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 29.2 kB in 0s (69.3 kB/s) 2024-05-21T17:19:01.870 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-05-21T17:19:01.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3827-g99470715-1jammy cephadm=19.0.0-3827-g99470715-1jammy ceph-mds=19.0.0-3827-g99470715-1jammy ceph-mgr=19.0.0-3827-g99470715-1jammy ceph-common=19.0.0-3827-g99470715-1jammy ceph-fuse=19.0.0-3827-g99470715-1jammy ceph-test=19.0.0-3827-g99470715-1jammy ceph-volume=19.0.0-3827-g99470715-1jammy radosgw=19.0.0-3827-g99470715-1jammy python3-rados=19.0.0-3827-g99470715-1jammy python3-rgw=19.0.0-3827-g99470715-1jammy python3-cephfs=19.0.0-3827-g99470715-1jammy python3-rbd=19.0.0-3827-g99470715-1jammy libcephfs2=19.0.0-3827-g99470715-1jammy libcephfs-dev=19.0.0-3827-g99470715-1jammy librados2=19.0.0-3827-g99470715-1jammy librbd1=19.0.0-3827-g99470715-1jammy rbd-fuse=19.0.0-3827-g99470715-1jammy 2024-05-21T17:19:01.923 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-05-21T17:19:01.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3827-g99470715-1jammy cephadm=19.0.0-3827-g99470715-1jammy ceph-mds=19.0.0-3827-g99470715-1jammy ceph-mgr=19.0.0-3827-g99470715-1jammy ceph-common=19.0.0-3827-g99470715-1jammy ceph-fuse=19.0.0-3827-g99470715-1jammy ceph-test=19.0.0-3827-g99470715-1jammy ceph-volume=19.0.0-3827-g99470715-1jammy radosgw=19.0.0-3827-g99470715-1jammy python3-rados=19.0.0-3827-g99470715-1jammy python3-rgw=19.0.0-3827-g99470715-1jammy python3-cephfs=19.0.0-3827-g99470715-1jammy python3-rbd=19.0.0-3827-g99470715-1jammy libcephfs2=19.0.0-3827-g99470715-1jammy libcephfs-dev=19.0.0-3827-g99470715-1jammy librados2=19.0.0-3827-g99470715-1jammy librbd1=19.0.0-3827-g99470715-1jammy rbd-fuse=19.0.0-3827-g99470715-1jammy 2024-05-21T17:19:01.947 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-05-21T17:19:02.003 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-05-21T17:19:02.147 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-05-21T17:19:02.148 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-05-21T17:19:02.204 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2024-05-21T17:19:02.204 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2024-05-21T17:19:02.335 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-05-21T17:19:02.335 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-21T17:19:02.335 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-21T17:19:02.335 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-21T17:19:02.336 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-21T17:19:02.336 INFO:teuthology.orchestra.run.smithi203.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-21T17:19:02.336 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-21T17:19:02.337 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-05-21T17:19:02.337 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-21T17:19:02.337 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-21T17:19:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-21T17:19:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-21T17:19:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-21T17:19:02.338 INFO:teuthology.orchestra.run.smithi203.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python-asyncssh-doc python-babel-localedata python3-asyncssh python3-babel 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt python3-cachetools python3-ceph-argparse python3-ceph-common 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-markupsafe 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort python3-openssl python3-pluggy python3-portend 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules python3-pygments python3-pytest python3-repoze.lru 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-wcwidth 2024-05-21T17:19:02.339 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2024-05-21T17:19:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: socat unzip xmlstarlet zip 2024-05-21T17:19:02.340 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-05-21T17:19:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: python3-influxdb readline-doc python-jinja2-doc python-natsort-doc 2024-05-21T17:19:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: python-openssl-doc python3-openssl-dbg python-psutil-doc subversion 2024-05-21T17:19:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: python-pygments-doc python3-socks python-requests-doc python3-paste 2024-05-21T17:19:02.340 INFO:teuthology.orchestra.run.smithi203.stdout: python3-dap python-sklearn-doc ipython3 python-webob-doc 2024-05-21T17:19:02.340 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-05-21T17:19:02.341 INFO:teuthology.orchestra.run.smithi203.stdout: btrfs-tools 2024-05-21T17:19:02.406 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages were automatically installed and are no longer required: 2024-05-21T17:19:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-21T17:19:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-21T17:19:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-21T17:19:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-21T17:19:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-21T17:19:02.408 INFO:teuthology.orchestra.run.smithi003.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-21T17:19:02.408 INFO:teuthology.orchestra.run.smithi003.stdout:The following additional packages will be installed: 2024-05-21T17:19:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-21T17:19:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-21T17:19:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-21T17:19:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: python-asyncssh-doc python-babel-localedata python3-asyncssh python3-babel 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt python3-cachetools python3-ceph-argparse python3-ceph-common 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-markupsafe 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort python3-openssl python3-pluggy python3-portend 2024-05-21T17:19:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules python3-pygments python3-pytest python3-repoze.lru 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-wcwidth 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: socat unzip xmlstarlet zip 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout:Suggested packages: 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: python3-influxdb readline-doc python-jinja2-doc python-natsort-doc 2024-05-21T17:19:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: python-openssl-doc python3-openssl-dbg python-psutil-doc subversion 2024-05-21T17:19:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: python-pygments-doc python3-socks python-requests-doc python3-paste 2024-05-21T17:19:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: python3-dap python-sklearn-doc ipython3 python-webob-doc 2024-05-21T17:19:02.412 INFO:teuthology.orchestra.run.smithi003.stdout:Recommended packages: 2024-05-21T17:19:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: btrfs-tools 2024-05-21T17:19:02.435 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-05-21T17:19:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-21T17:19:02.436 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-21T17:19:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-21T17:19:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-21T17:19:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-21T17:19:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-21T17:19:02.437 INFO:teuthology.orchestra.run.smithi203.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python-babel-localedata python3-asyncssh python3-babel python3-bcrypt 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-markupsafe 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort python3-openssl python3-pluggy python3-portend 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules python3-pygments python3-pytest python3-rados 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd python3-repoze.lru python3-requests python3-requests-oauthlib 2024-05-21T17:19:02.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw python3-routes python3-rsa python3-simplejson python3-sklearn 2024-05-21T17:19:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: python3-sklearn-lib python3-tempora python3-threadpoolctl python3-toml 2024-05-21T17:19:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 python3-wcwidth python3-webob python3-websocket 2024-05-21T17:19:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse socat unzip 2024-05-21T17:19:02.439 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet zip 2024-05-21T17:19:02.518 INFO:teuthology.orchestra.run.smithi003.stdout:The following NEW packages will be installed: 2024-05-21T17:19:02.518 INFO:teuthology.orchestra.run.smithi003.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-21T17:19:02.518 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-21T17:19:02.518 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-21T17:19:02.518 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-21T17:19:02.518 INFO:teuthology.orchestra.run.smithi003.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-21T17:19:02.518 INFO:teuthology.orchestra.run.smithi003.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python-babel-localedata python3-asyncssh python3-babel python3-bcrypt 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-markupsafe 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort python3-openssl python3-pluggy python3-portend 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules python3-pygments python3-pytest python3-rados 2024-05-21T17:19:02.520 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd python3-repoze.lru python3-requests python3-requests-oauthlib 2024-05-21T17:19:02.521 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw python3-routes python3-rsa python3-simplejson python3-sklearn 2024-05-21T17:19:02.521 INFO:teuthology.orchestra.run.smithi003.stdout: python3-sklearn-lib python3-tempora python3-threadpoolctl python3-toml 2024-05-21T17:19:02.521 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 python3-wcwidth python3-webob python3-websocket 2024-05-21T17:19:02.521 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse socat unzip 2024-05-21T17:19:02.521 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet zip 2024-05-21T17:19:02.626 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 104 newly installed, 0 to remove and 307 not upgraded. 2024-05-21T17:19:02.626 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 188 MB of archives. 2024-05-21T17:19:02.626 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 858 MB of additional disk space will be used. 2024-05-21T17:19:02.626 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-21T17:19:02.663 INFO:teuthology.orchestra.run.smithi003.stdout:0 upgraded, 104 newly installed, 0 to remove and 307 not upgraded. 2024-05-21T17:19:02.664 INFO:teuthology.orchestra.run.smithi003.stdout:Need to get 188 MB of archives. 2024-05-21T17:19:02.664 INFO:teuthology.orchestra.run.smithi003.stdout:After this operation, 858 MB of additional disk space will be used. 2024-05-21T17:19:02.664 INFO:teuthology.orchestra.run.smithi003.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3827-g99470715-1jammy [3,557 kB] 2024-05-21T17:19:02.709 INFO:teuthology.orchestra.run.smithi003.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-21T17:19:02.978 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3827-g99470715-1jammy [3,226 kB] 2024-05-21T17:19:03.000 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3827-g99470715-1jammy [3,557 kB] 2024-05-21T17:19:03.112 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3827-g99470715-1jammy [713 kB] 2024-05-21T17:19:03.135 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-21T17:19:03.145 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3827-g99470715-1jammy [357 kB] 2024-05-21T17:19:03.147 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-21T17:19:03.153 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3827-g99470715-1jammy [32.8 kB] 2024-05-21T17:19:03.154 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-21T17:19:03.154 INFO:teuthology.orchestra.run.smithi003.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3827-g99470715-1jammy [180 kB] 2024-05-21T17:19:03.165 INFO:teuthology.orchestra.run.smithi003.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3827-g99470715-1jammy [67.4 kB] 2024-05-21T17:19:03.167 INFO:teuthology.orchestra.run.smithi003.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3827-g99470715-1jammy [333 kB] 2024-05-21T17:19:03.174 INFO:teuthology.orchestra.run.smithi003.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3827-g99470715-1jammy [7,880 kB] 2024-05-21T17:19:03.180 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-21T17:19:03.229 INFO:teuthology.orchestra.run.smithi003.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-21T17:19:03.286 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-21T17:19:03.434 INFO:teuthology.orchestra.run.smithi003.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3827-g99470715-1jammy [112 kB] 2024-05-21T17:19:03.436 INFO:teuthology.orchestra.run.smithi003.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3827-g99470715-1jammy [466 kB] 2024-05-21T17:19:03.451 INFO:teuthology.orchestra.run.smithi003.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3827-g99470715-1jammy [27.8 MB] 2024-05-21T17:19:03.459 INFO:teuthology.orchestra.run.smithi003.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-21T17:19:03.471 INFO:teuthology.orchestra.run.smithi003.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-21T17:19:03.501 INFO:teuthology.orchestra.run.smithi003.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-21T17:19:03.510 INFO:teuthology.orchestra.run.smithi003.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-21T17:19:03.511 INFO:teuthology.orchestra.run.smithi003.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-21T17:19:03.512 INFO:teuthology.orchestra.run.smithi003.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-21T17:19:03.531 INFO:teuthology.orchestra.run.smithi003.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-21T17:19:03.536 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-21T17:19:03.547 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-21T17:19:03.577 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-21T17:19:03.588 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-21T17:19:03.589 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-21T17:19:03.591 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-21T17:19:03.614 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-21T17:19:03.700 INFO:teuthology.orchestra.run.smithi003.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-21T17:19:03.815 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-21T17:19:04.112 INFO:teuthology.orchestra.run.smithi003.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-21T17:19:04.146 INFO:teuthology.orchestra.run.smithi003.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-21T17:19:04.148 INFO:teuthology.orchestra.run.smithi003.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-21T17:19:04.229 INFO:teuthology.orchestra.run.smithi003.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-21T17:19:04.232 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-21T17:19:04.265 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-21T17:19:04.268 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-21T17:19:04.280 INFO:teuthology.orchestra.run.smithi003.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-21T17:19:04.304 INFO:teuthology.orchestra.run.smithi003.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-21T17:19:04.310 INFO:teuthology.orchestra.run.smithi003.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-21T17:19:04.312 INFO:teuthology.orchestra.run.smithi003.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-21T17:19:04.326 INFO:teuthology.orchestra.run.smithi003.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-21T17:19:04.326 INFO:teuthology.orchestra.run.smithi003.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-21T17:19:04.348 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-21T17:19:04.401 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-21T17:19:04.423 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-21T17:19:04.430 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-21T17:19:04.431 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-21T17:19:04.444 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-21T17:19:04.445 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-21T17:19:04.491 INFO:teuthology.orchestra.run.smithi003.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-21T17:19:04.509 INFO:teuthology.orchestra.run.smithi003.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3827-g99470715-1jammy [5,093 kB] 2024-05-21T17:19:04.590 INFO:teuthology.orchestra.run.smithi003.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3827-g99470715-1jammy [230 kB] 2024-05-21T17:19:04.591 INFO:teuthology.orchestra.run.smithi003.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-21T17:19:04.593 INFO:teuthology.orchestra.run.smithi003.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3827-g99470715-1jammy [124 kB] 2024-05-21T17:19:04.594 INFO:teuthology.orchestra.run.smithi003.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3827-g99470715-1jammy [1,545 kB] 2024-05-21T17:19:04.614 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-21T17:19:04.619 INFO:teuthology.orchestra.run.smithi003.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-21T17:19:04.622 INFO:teuthology.orchestra.run.smithi003.stdout:Get:42 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3827-g99470715-1jammy [6,260 kB] 2024-05-21T17:19:04.637 INFO:teuthology.orchestra.run.smithi003.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-21T17:19:04.681 INFO:teuthology.orchestra.run.smithi003.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-21T17:19:04.700 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-21T17:19:04.729 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-21T17:19:04.747 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-21T17:19:04.793 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-21T17:19:04.830 INFO:teuthology.orchestra.run.smithi003.stdout:Get:45 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3827-g99470715-1jammy [22.8 MB] 2024-05-21T17:19:04.896 INFO:teuthology.orchestra.run.smithi003.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-21T17:19:04.911 INFO:teuthology.orchestra.run.smithi003.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-21T17:19:04.962 INFO:teuthology.orchestra.run.smithi003.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-05-21T17:19:04.976 INFO:teuthology.orchestra.run.smithi003.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-21T17:19:04.998 INFO:teuthology.orchestra.run.smithi003.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-21T17:19:05.007 INFO:teuthology.orchestra.run.smithi003.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-21T17:19:05.011 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-21T17:19:05.026 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-21T17:19:05.099 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-05-21T17:19:05.111 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-21T17:19:05.132 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-21T17:19:05.144 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-21T17:19:05.176 INFO:teuthology.orchestra.run.smithi003.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-21T17:19:05.316 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-21T17:19:05.430 INFO:teuthology.orchestra.run.smithi003.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-21T17:19:05.571 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-21T17:19:05.669 INFO:teuthology.orchestra.run.smithi003.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-21T17:19:05.670 INFO:teuthology.orchestra.run.smithi003.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3827-g99470715-1jammy [14.0 kB] 2024-05-21T17:19:05.671 INFO:teuthology.orchestra.run.smithi003.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3827-g99470715-1jammy [870 kB] 2024-05-21T17:19:05.688 INFO:teuthology.orchestra.run.smithi003.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3827-g99470715-1jammy [2,441 kB] 2024-05-21T17:19:05.730 INFO:teuthology.orchestra.run.smithi003.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3827-g99470715-1jammy [792 kB] 2024-05-21T17:19:05.740 INFO:teuthology.orchestra.run.smithi003.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3827-g99470715-1jammy [149 kB] 2024-05-21T17:19:05.742 INFO:teuthology.orchestra.run.smithi003.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3827-g99470715-1jammy [2,222 kB] 2024-05-21T17:19:05.801 INFO:teuthology.orchestra.run.smithi003.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3827-g99470715-1jammy [8,625 kB] 2024-05-21T17:19:05.816 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-21T17:19:06.020 INFO:teuthology.orchestra.run.smithi003.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-21T17:19:06.022 INFO:teuthology.orchestra.run.smithi003.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-21T17:19:06.023 INFO:teuthology.orchestra.run.smithi003.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-21T17:19:06.025 INFO:teuthology.orchestra.run.smithi003.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-21T17:19:06.026 INFO:teuthology.orchestra.run.smithi003.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-21T17:19:06.028 INFO:teuthology.orchestra.run.smithi003.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-21T17:19:06.089 INFO:teuthology.orchestra.run.smithi003.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3827-g99470715-1jammy [14.2 kB] 2024-05-21T17:19:06.089 INFO:teuthology.orchestra.run.smithi003.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3827-g99470715-1jammy [54.7 MB] 2024-05-21T17:19:06.091 INFO:teuthology.orchestra.run.smithi003.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-21T17:19:06.095 INFO:teuthology.orchestra.run.smithi003.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-21T17:19:06.231 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-21T17:19:06.233 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-21T17:19:06.234 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-21T17:19:06.236 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-21T17:19:06.237 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-21T17:19:06.239 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-21T17:19:06.261 INFO:teuthology.orchestra.run.smithi003.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-21T17:19:06.298 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-21T17:19:06.302 INFO:teuthology.orchestra.run.smithi203.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-21T17:19:06.469 INFO:teuthology.orchestra.run.smithi203.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-21T17:19:06.958 INFO:teuthology.orchestra.run.smithi003.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-21T17:19:06.959 INFO:teuthology.orchestra.run.smithi003.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-21T17:19:06.960 INFO:teuthology.orchestra.run.smithi003.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-21T17:19:06.962 INFO:teuthology.orchestra.run.smithi003.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-21T17:19:06.963 INFO:teuthology.orchestra.run.smithi003.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-21T17:19:06.964 INFO:teuthology.orchestra.run.smithi003.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-21T17:19:06.965 INFO:teuthology.orchestra.run.smithi003.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-21T17:19:06.966 INFO:teuthology.orchestra.run.smithi003.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-21T17:19:06.975 INFO:teuthology.orchestra.run.smithi003.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-21T17:19:07.039 INFO:teuthology.orchestra.run.smithi003.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-21T17:19:07.208 INFO:teuthology.orchestra.run.smithi003.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-21T17:19:07.220 INFO:teuthology.orchestra.run.smithi203.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-21T17:19:07.221 INFO:teuthology.orchestra.run.smithi203.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-21T17:19:07.223 INFO:teuthology.orchestra.run.smithi203.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-21T17:19:07.224 INFO:teuthology.orchestra.run.smithi203.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-21T17:19:07.228 INFO:teuthology.orchestra.run.smithi203.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-21T17:19:07.229 INFO:teuthology.orchestra.run.smithi203.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-21T17:19:07.230 INFO:teuthology.orchestra.run.smithi203.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-21T17:19:07.231 INFO:teuthology.orchestra.run.smithi203.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-21T17:19:07.248 INFO:teuthology.orchestra.run.smithi203.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-21T17:19:07.302 INFO:teuthology.orchestra.run.smithi203.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-21T17:19:07.475 INFO:teuthology.orchestra.run.smithi203.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-21T17:19:07.476 INFO:teuthology.orchestra.run.smithi003.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-21T17:19:07.725 INFO:teuthology.orchestra.run.smithi003.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-21T17:19:07.743 INFO:teuthology.orchestra.run.smithi203.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-21T17:19:07.786 INFO:teuthology.orchestra.run.smithi003.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-21T17:19:07.799 INFO:teuthology.orchestra.run.smithi003.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-21T17:19:07.805 INFO:teuthology.orchestra.run.smithi003.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-21T17:19:07.863 INFO:teuthology.orchestra.run.smithi003.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-21T17:19:07.868 INFO:teuthology.orchestra.run.smithi003.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-21T17:19:07.894 INFO:teuthology.orchestra.run.smithi003.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-21T17:19:07.894 INFO:teuthology.orchestra.run.smithi003.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-21T17:19:07.897 INFO:teuthology.orchestra.run.smithi003.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-21T17:19:07.990 INFO:teuthology.orchestra.run.smithi203.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-21T17:19:08.051 INFO:teuthology.orchestra.run.smithi203.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-21T17:19:08.064 INFO:teuthology.orchestra.run.smithi203.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-21T17:19:08.070 INFO:teuthology.orchestra.run.smithi203.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-21T17:19:08.071 INFO:teuthology.orchestra.run.smithi003.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-21T17:19:08.129 INFO:teuthology.orchestra.run.smithi203.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-21T17:19:08.132 INFO:teuthology.orchestra.run.smithi003.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3827-g99470715-1jammy [124 kB] 2024-05-21T17:19:08.133 INFO:teuthology.orchestra.run.smithi203.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-21T17:19:08.136 INFO:teuthology.orchestra.run.smithi003.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3827-g99470715-1jammy [38.9 kB] 2024-05-21T17:19:08.136 INFO:teuthology.orchestra.run.smithi003.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3827-g99470715-1jammy [15.7 MB] 2024-05-21T17:19:08.161 INFO:teuthology.orchestra.run.smithi203.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-21T17:19:08.162 INFO:teuthology.orchestra.run.smithi203.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-21T17:19:08.164 INFO:teuthology.orchestra.run.smithi203.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-21T17:19:08.367 INFO:teuthology.orchestra.run.smithi203.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-21T17:19:08.381 INFO:teuthology.orchestra.run.smithi003.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-21T17:19:08.597 INFO:teuthology.orchestra.run.smithi003.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3827-g99470715-1jammy [91.4 kB] 2024-05-21T17:19:08.666 INFO:teuthology.orchestra.run.smithi203.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-21T17:19:08.690 INFO:teuthology.orchestra.run.smithi003.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-21T17:19:08.691 INFO:teuthology.orchestra.run.smithi003.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-21T17:19:08.719 INFO:teuthology.orchestra.run.smithi003.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-21T17:19:08.726 INFO:teuthology.orchestra.run.smithi003.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-21T17:19:08.736 INFO:teuthology.orchestra.run.smithi003.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-21T17:19:08.973 INFO:teuthology.orchestra.run.smithi203.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-21T17:19:08.975 INFO:teuthology.orchestra.run.smithi203.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-21T17:19:09.003 INFO:teuthology.orchestra.run.smithi203.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-21T17:19:09.010 INFO:teuthology.orchestra.run.smithi203.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-21T17:19:09.019 INFO:teuthology.orchestra.run.smithi203.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-21T17:19:09.236 INFO:teuthology.orchestra.run.smithi203.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3827-g99470715-1jammy [3,226 kB] 2024-05-21T17:19:09.433 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 188 MB in 6s (29.7 MB/s) 2024-05-21T17:19:09.601 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-21T17:19:09.704 INFO:teuthology.orchestra.run.smithi003.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-05-21T17:19:09.710 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-21T17:19:09.792 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-21T17:19:10.120 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-21T17:19:10.147 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-21T17:19:10.362 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-21T17:19:10.641 INFO:teuthology.orchestra.run.smithi203.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3827-g99470715-1jammy [713 kB] 2024-05-21T17:19:10.656 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-21T17:19:10.683 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-21T17:19:10.714 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-21T17:19:10.857 INFO:teuthology.orchestra.run.smithi203.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3827-g99470715-1jammy [357 kB] 2024-05-21T17:19:10.963 INFO:teuthology.orchestra.run.smithi203.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3827-g99470715-1jammy [32.8 kB] 2024-05-21T17:19:10.964 INFO:teuthology.orchestra.run.smithi203.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3827-g99470715-1jammy [180 kB] 2024-05-21T17:19:11.002 INFO:teuthology.orchestra.run.smithi203.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3827-g99470715-1jammy [67.4 kB] 2024-05-21T17:19:11.034 INFO:teuthology.orchestra.run.smithi203.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3827-g99470715-1jammy [333 kB] 2024-05-21T17:19:11.109 INFO:teuthology.orchestra.run.smithi203.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3827-g99470715-1jammy [7,880 kB] 2024-05-21T17:19:11.143 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-21T17:19:11.170 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-21T17:19:11.217 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:19:11.629 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-21T17:19:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-21T17:19:11.696 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:19:11.966 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-21T17:19:11.993 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-21T17:19:12.024 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:19:12.343 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-21T17:19:12.370 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-21T17:19:12.410 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-21T17:19:12.601 INFO:teuthology.orchestra.run.smithi203.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3827-g99470715-1jammy [112 kB] 2024-05-21T17:19:12.626 INFO:teuthology.orchestra.run.smithi203.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3827-g99470715-1jammy [466 kB] 2024-05-21T17:19:12.696 INFO:teuthology.orchestra.run.smithi203.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3827-g99470715-1jammy [27.8 MB] 2024-05-21T17:19:12.730 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librados2. 2024-05-21T17:19:12.757 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../007-librados2_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:12.796 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librados2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:13.241 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librbd1. 2024-05-21T17:19:13.267 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../008-librbd1_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:13.307 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librbd1 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:13.735 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libcephfs2. 2024-05-21T17:19:13.762 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:13.793 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libcephfs2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:14.079 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rados. 2024-05-21T17:19:14.095 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:14.129 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rados (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:14.423 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-21T17:19:14.450 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:14.490 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-ceph-argparse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:14.742 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cephfs. 2024-05-21T17:19:14.769 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:14.809 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cephfs (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:15.120 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-21T17:19:15.147 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:15.187 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-ceph-common (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:15.346 INFO:teuthology.orchestra.run.smithi203.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3827-g99470715-1jammy [5,093 kB] 2024-05-21T17:19:15.498 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-21T17:19:15.525 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-21T17:19:15.565 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-21T17:19:15.596 INFO:teuthology.orchestra.run.smithi203.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3827-g99470715-1jammy [230 kB] 2024-05-21T17:19:15.603 INFO:teuthology.orchestra.run.smithi203.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3827-g99470715-1jammy [124 kB] 2024-05-21T17:19:15.605 INFO:teuthology.orchestra.run.smithi203.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3827-g99470715-1jammy [1,545 kB] 2024-05-21T17:19:15.682 INFO:teuthology.orchestra.run.smithi203.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3827-g99470715-1jammy [6,260 kB] 2024-05-21T17:19:15.842 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-prettytable. 2024-05-21T17:19:15.869 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-21T17:19:15.900 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-21T17:19:16.005 INFO:teuthology.orchestra.run.smithi203.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3827-g99470715-1jammy [22.8 MB] 2024-05-21T17:19:16.153 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rbd. 2024-05-21T17:19:16.169 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:16.211 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rbd (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:16.548 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-21T17:19:16.574 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-21T17:19:16.614 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-21T17:19:16.892 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-21T17:19:16.908 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-21T17:19:16.950 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-21T17:19:17.211 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-21T17:19:17.213 INFO:teuthology.orchestra.run.smithi203.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3827-g99470715-1jammy [14.0 kB] 2024-05-21T17:19:17.214 INFO:teuthology.orchestra.run.smithi203.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3827-g99470715-1jammy [870 kB] 2024-05-21T17:19:17.227 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-21T17:19:17.230 INFO:teuthology.orchestra.run.smithi203.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3827-g99470715-1jammy [2,441 kB] 2024-05-21T17:19:17.261 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-21T17:19:17.291 INFO:teuthology.orchestra.run.smithi203.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3827-g99470715-1jammy [792 kB] 2024-05-21T17:19:17.322 INFO:teuthology.orchestra.run.smithi203.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3827-g99470715-1jammy [149 kB] 2024-05-21T17:19:17.326 INFO:teuthology.orchestra.run.smithi203.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3827-g99470715-1jammy [2,222 kB] 2024-05-21T17:19:17.400 INFO:teuthology.orchestra.run.smithi203.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3827-g99470715-1jammy [8,625 kB] 2024-05-21T17:19:17.584 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package lua5.1. 2024-05-21T17:19:17.611 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-21T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-21T17:19:17.688 INFO:teuthology.orchestra.run.smithi203.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3827-g99470715-1jammy [14.2 kB] 2024-05-21T17:19:17.689 INFO:teuthology.orchestra.run.smithi203.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3827-g99470715-1jammy [54.7 MB] 2024-05-21T17:19:17.967 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package lua-any. 2024-05-21T17:19:17.983 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-21T17:19:18.025 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-21T17:19:18.261 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package zip. 2024-05-21T17:19:18.277 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-21T17:19:18.311 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking zip (3.0-12build2) ... 2024-05-21T17:19:18.581 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package unzip. 2024-05-21T17:19:18.608 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-21T17:19:18.648 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-21T17:19:19.268 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package luarocks. 2024-05-21T17:19:19.295 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-21T17:19:19.335 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-21T17:19:19.701 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librgw2. 2024-05-21T17:19:19.728 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../025-librgw2_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:19.767 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librgw2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:19.808 INFO:teuthology.orchestra.run.smithi203.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3827-g99470715-1jammy [124 kB] 2024-05-21T17:19:19.810 INFO:teuthology.orchestra.run.smithi203.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3827-g99470715-1jammy [38.9 kB] 2024-05-21T17:19:19.811 INFO:teuthology.orchestra.run.smithi203.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3827-g99470715-1jammy [15.7 MB] 2024-05-21T17:19:20.320 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rgw. 2024-05-21T17:19:20.336 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:20.370 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rgw (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:20.577 INFO:teuthology.orchestra.run.smithi203.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-05-20-1050/99470715356be3e23220ec4a5eacc2bc4dfde33f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3827-g99470715-1jammy [91.4 kB] 2024-05-21T17:19:20.697 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-21T17:19:20.725 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-21T17:19:20.764 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-21T17:19:21.100 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libradosstriper1. 2024-05-21T17:19:21.127 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 188 MB in 18s (10.4 MB/s) 2024-05-21T17:19:21.167 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libradosstriper1 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-21T17:19:21.391 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-05-21T17:19:21.396 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-21T17:19:21.485 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-21T17:19:21.529 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-common. 2024-05-21T17:19:21.556 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:21.586 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-common (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:21.846 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-21T17:19:21.873 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-21T17:19:21.913 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-21T17:19:22.216 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-21T17:19:22.242 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-21T17:19:22.282 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-21T17:19:22.719 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-21T17:19:22.745 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-21T17:19:22.794 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:19:23.058 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-base. 2024-05-21T17:19:23.085 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:23.134 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-base (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:23.214 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-21T17:19:23.240 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-21T17:19:23.297 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:19:23.675 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-21T17:19:23.702 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-21T17:19:23.742 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:19:23.762 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-21T17:19:23.789 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-21T17:19:23.829 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-21T17:19:24.078 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-21T17:19:24.104 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-21T17:19:24.107 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cheroot. 2024-05-21T17:19:24.134 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-21T17:19:24.136 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-21T17:19:24.165 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-21T17:19:24.448 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librados2. 2024-05-21T17:19:24.474 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../007-librados2_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:24.514 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librados2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:24.685 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-21T17:19:24.712 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-21T17:19:24.751 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-21T17:19:24.984 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librbd1. 2024-05-21T17:19:25.010 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../008-librbd1_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:25.050 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librbd1 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:25.063 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-21T17:19:25.090 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-21T17:19:25.129 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-21T17:19:25.432 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-21T17:19:25.459 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-21T17:19:25.486 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs2. 2024-05-21T17:19:25.498 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-21T17:19:25.512 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:25.552 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:25.776 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-tempora. 2024-05-21T17:19:25.803 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-21T17:19:25.830 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rados. 2024-05-21T17:19:25.843 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-21T17:19:25.845 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:25.880 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rados (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:26.121 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-portend. 2024-05-21T17:19:26.148 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-21T17:19:26.187 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-21T17:19:26.341 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-21T17:19:26.368 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:26.408 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-argparse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:26.490 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-21T17:19:26.517 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-21T17:19:26.557 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-21T17:19:26.669 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cephfs. 2024-05-21T17:19:26.695 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:26.735 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cephfs (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:26.859 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-21T17:19:26.887 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-21T17:19:26.926 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-21T17:19:27.046 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-21T17:19:27.073 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:27.129 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-common (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:27.271 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-natsort. 2024-05-21T17:19:27.298 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-21T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-21T17:19:27.441 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-21T17:19:27.456 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-21T17:19:27.491 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-21T17:19:27.666 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-certifi. 2024-05-21T17:19:27.693 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../041-python3-certifi_2020.6.20-1_all.deb ... 2024-05-21T17:19:27.732 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-21T17:19:27.752 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-prettytable. 2024-05-21T17:19:27.770 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-21T17:19:27.802 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-21T17:19:28.035 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-idna. 2024-05-21T17:19:28.062 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../042-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-05-21T17:19:28.090 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rbd. 2024-05-21T17:19:28.102 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-05-21T17:19:28.116 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:28.157 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rbd (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:28.405 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-urllib3. 2024-05-21T17:19:28.432 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../043-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-21T17:19:28.468 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-21T17:19:28.471 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-21T17:19:28.494 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-21T17:19:28.535 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-21T17:19:28.766 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-requests. 2024-05-21T17:19:28.770 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-21T17:19:28.786 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-21T17:19:28.794 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../044-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-21T17:19:28.829 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-21T17:19:28.832 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-21T17:19:29.093 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-21T17:19:29.109 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../045-ceph-mgr-modules-core_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-21T17:19:29.139 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-21T17:19:29.151 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:29.182 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-21T17:19:29.498 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-21T17:19:29.514 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../046-libsqlite3-mod-ceph_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:29.548 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:29.677 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua5.1. 2024-05-21T17:19:29.704 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-21T17:19:29.744 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-21T17:19:29.842 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-21T17:19:29.869 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../047-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-21T17:19:29.909 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-21T17:19:30.031 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-any. 2024-05-21T17:19:30.046 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-21T17:19:30.089 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-21T17:19:30.212 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-openssl. 2024-05-21T17:19:30.239 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../048-python3-openssl_21.0.0-1_all.deb ... 2024-05-21T17:19:30.278 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-21T17:19:30.342 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package zip. 2024-05-21T17:19:30.357 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-21T17:19:30.400 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking zip (3.0-12build2) ... 2024-05-21T17:19:30.615 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr. 2024-05-21T17:19:30.642 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../049-ceph-mgr_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:30.681 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:30.762 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package unzip. 2024-05-21T17:19:30.789 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-21T17:19:30.829 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-21T17:19:31.034 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mon. 2024-05-21T17:19:31.061 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../050-ceph-mon_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:31.101 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mon (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:31.208 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package luarocks. 2024-05-21T17:19:31.223 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-21T17:19:31.258 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-21T17:19:31.650 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librgw2. 2024-05-21T17:19:31.676 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../025-librgw2_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:31.713 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-osd. 2024-05-21T17:19:31.716 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librgw2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:31.740 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../051-ceph-osd_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:31.779 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-osd (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:32.344 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rgw. 2024-05-21T17:19:32.359 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:32.402 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rgw (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:32.714 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-21T17:19:32.740 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-21T17:19:32.772 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-21T17:19:32.866 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph. 2024-05-21T17:19:32.882 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../052-ceph_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:32.924 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:33.075 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libradosstriper1. 2024-05-21T17:19:33.102 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:33.142 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libradosstriper1 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:33.194 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-fuse. 2024-05-21T17:19:33.220 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../053-ceph-fuse_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:33.252 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-fuse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:33.570 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-common. 2024-05-21T17:19:33.597 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:33.613 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mds. 2024-05-21T17:19:33.637 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-common (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:33.640 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../054-ceph-mds_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:33.680 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mds (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:34.058 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package cephadm. 2024-05-21T17:19:34.085 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../055-cephadm_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:34.124 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking cephadm (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:34.427 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-21T17:19:34.454 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../056-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-21T17:19:34.485 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-21T17:19:34.889 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-21T17:19:34.916 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../057-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-21T17:19:34.955 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-21T17:19:35.251 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-base. 2024-05-21T17:19:35.278 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:35.327 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-base (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-babel. 2024-05-21T17:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../058-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-21T17:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-21T17:19:36.915 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-21T17:19:36.942 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-21T17:19:36.981 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-21T17:19:36.997 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-21T17:19:37.024 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../059-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-21T17:19:37.063 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-21T17:19:37.251 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cheroot. 2024-05-21T17:19:37.266 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-21T17:19:37.300 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-21T17:19:37.349 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jinja2. 2024-05-21T17:19:37.365 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../060-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-21T17:19:37.407 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-21T17:19:37.606 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-21T17:19:37.630 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-21T17:19:37.670 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-21T17:19:37.786 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-21T17:19:37.813 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../061-ceph-mgr-cephadm_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:37.852 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:37.973 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-21T17:19:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-21T17:19:38.040 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-21T17:19:38.206 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-21T17:19:38.233 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../062-python3-repoze.lru_0.7-2_all.deb ... 2024-05-21T17:19:38.272 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-21T17:19:38.326 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-21T17:19:38.353 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-21T17:19:38.393 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-21T17:19:38.550 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-routes. 2024-05-21T17:19:38.567 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../063-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-21T17:19:38.608 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-21T17:19:38.671 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-tempora. 2024-05-21T17:19:38.697 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-21T17:19:38.737 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-21T17:19:38.945 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-21T17:19:38.972 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../064-ceph-mgr-dashboard_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:39.011 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:39.066 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-portend. 2024-05-21T17:19:39.092 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-21T17:19:39.132 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-21T17:19:39.435 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-21T17:19:39.462 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-21T17:19:39.502 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-21T17:19:39.772 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-21T17:19:39.798 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-21T17:19:39.830 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-21T17:19:40.151 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-natsort. 2024-05-21T17:19:40.175 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-21T17:19:40.177 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-21T17:19:40.192 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../065-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-21T17:19:40.217 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-21T17:19:40.225 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-21T17:19:40.529 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-certifi. 2024-05-21T17:19:40.555 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../041-python3-certifi_2020.6.20-1_all.deb ... 2024-05-21T17:19:40.595 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-21T17:19:40.687 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-joblib. 2024-05-21T17:19:40.703 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../066-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-21T17:19:40.745 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-21T17:19:40.940 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-idna. 2024-05-21T17:19:40.967 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../042-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-05-21T17:19:41.007 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-05-21T17:19:41.124 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-21T17:19:41.151 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../067-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-21T17:19:41.190 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-21T17:19:41.302 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-urllib3. 2024-05-21T17:19:41.328 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../043-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-21T17:19:41.360 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-21T17:19:41.460 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-sklearn. 2024-05-21T17:19:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../068-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-21T17:19:41.518 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-21T17:19:41.647 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests. 2024-05-21T17:19:41.673 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../044-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-21T17:19:41.705 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-21T17:19:41.925 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-21T17:19:41.941 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../045-ceph-mgr-modules-core_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:41.983 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:42.235 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-21T17:19:42.263 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:42.301 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:42.354 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-21T17:19:42.381 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../046-libsqlite3-mod-ceph_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:42.421 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:42.749 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-21T17:19:42.776 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../047-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-21T17:19:42.816 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-21T17:19:43.102 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-openssl. 2024-05-21T17:19:43.128 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../048-python3-openssl_21.0.0-1_all.deb ... 2024-05-21T17:19:43.160 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-21T17:19:43.204 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cachetools. 2024-05-21T17:19:43.232 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../070-python3-cachetools_5.0.0-1_all.deb ... 2024-05-21T17:19:43.262 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-21T17:19:43.497 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr. 2024-05-21T17:19:43.523 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../049-ceph-mgr_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:43.563 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:43.565 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-21T17:19:43.593 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-21T17:19:43.631 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-21T17:19:43.968 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-21T17:19:43.975 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mon. 2024-05-21T17:19:43.996 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-21T17:19:44.002 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../050-ceph-mon_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:44.034 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-21T17:19:44.041 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mon (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:44.387 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rsa. 2024-05-21T17:19:44.415 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../073-python3-rsa_4.8-1_all.deb ... 2024-05-21T17:19:44.462 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-21T17:19:44.620 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-osd. 2024-05-21T17:19:44.647 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../051-ceph-osd_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:44.678 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-osd (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:44.749 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-google-auth. 2024-05-21T17:19:44.765 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-3_all.deb ... 2024-05-21T17:19:44.799 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-21T17:19:45.268 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-21T17:19:45.296 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../075-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-21T17:19:45.326 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-21T17:19:45.629 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-websocket. 2024-05-21T17:19:45.657 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../076-python3-websocket_1.2.3-1_all.deb ... 2024-05-21T17:19:45.696 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-21T17:19:45.733 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph. 2024-05-21T17:19:45.759 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../052-ceph_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:45.799 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-21T17:19:46.019 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../077-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-21T17:19:46.049 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-21T17:19:46.119 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-fuse. 2024-05-21T17:19:46.145 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../053-ceph-fuse_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:46.185 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-fuse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:46.597 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mds. 2024-05-21T17:19:46.624 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../054-ceph-mds_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:46.654 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-21T17:19:46.663 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mds (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:46.682 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../078-ceph-mgr-k8sevents_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:46.722 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:47.042 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-21T17:19:47.050 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package cephadm. 2024-05-21T17:19:47.069 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../079-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-21T17:19:47.077 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../055-cephadm_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:47.108 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-21T17:19:47.117 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking cephadm (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:47.394 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-21T17:19:47.422 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../080-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-21T17:19:47.437 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-21T17:19:47.452 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-21T17:19:47.463 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../056-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-21T17:19:47.503 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-21T17:19:47.722 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package jq. 2024-05-21T17:19:47.749 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../081-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-21T17:19:47.788 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-21T17:19:47.823 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-21T17:19:47.838 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../057-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-21T17:19:47.881 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-21T17:19:48.041 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package socat. 2024-05-21T17:19:48.069 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../082-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-21T17:19:48.184 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-21T17:19:48.471 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package xmlstarlet. 2024-05-21T17:19:48.487 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../083-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-21T17:19:48.529 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-21T17:19:48.887 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-babel. 2024-05-21T17:19:48.892 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-test. 2024-05-21T17:19:48.908 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../084-ceph-test_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:48.914 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../058-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-21T17:19:48.942 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-test (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:48.945 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-21T17:19:49.257 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-21T17:19:49.284 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../059-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-21T17:19:49.323 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-21T17:19:49.626 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jinja2. 2024-05-21T17:19:49.653 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../060-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-21T17:19:49.693 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-21T17:19:50.038 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-21T17:19:50.065 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../061-ceph-mgr-cephadm_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:50.105 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:50.484 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-21T17:19:50.511 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../062-python3-repoze.lru_0.7-2_all.deb ... 2024-05-21T17:19:50.550 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-21T17:19:50.854 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-routes. 2024-05-21T17:19:50.881 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../063-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-21T17:19:50.920 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-21T17:19:51.274 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-21T17:19:51.301 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../064-ceph-mgr-dashboard_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:51.313 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-volume. 2024-05-21T17:19:51.340 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:51.341 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../085-ceph-volume_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:51.379 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-volume (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:51.683 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-21T17:19:51.711 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../086-libcephfs-dev_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:51.750 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libcephfs-dev (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:52.061 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-21T17:19:52.089 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../087-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-21T17:19:52.127 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-21T17:19:52.441 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-21T17:19:52.469 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../088-lua-sec_1.0.2-1_amd64.deb ... 2024-05-21T17:19:52.499 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-21T17:19:52.862 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package nvme-cli. 2024-05-21T17:19:52.889 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../089-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-21T17:19:52.928 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-21T17:19:52.946 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-21T17:19:52.961 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../065-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-21T17:19:52.996 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-21T17:19:53.367 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package pkg-config. 2024-05-21T17:19:53.394 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../090-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-21T17:19:53.433 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-21T17:19:53.508 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-joblib. 2024-05-21T17:19:53.535 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../066-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-21T17:19:53.575 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-21T17:19:53.694 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-21T17:19:53.710 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../091-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-21T17:19:53.752 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-21T17:19:53.946 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-21T17:19:53.972 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../067-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-21T17:19:54.012 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-21T17:19:54.164 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-21T17:19:54.192 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../092-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-21T17:19:54.238 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-21T17:19:54.298 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn. 2024-05-21T17:19:54.325 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../068-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-21T17:19:54.365 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-21T17:19:54.508 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pluggy. 2024-05-21T17:19:54.525 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../093-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-21T17:19:54.558 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-21T17:19:54.827 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-psutil. 2024-05-21T17:19:54.856 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../094-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-21T17:19:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-21T17:19:55.074 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-21T17:19:55.102 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:55.141 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:55.188 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-py. 2024-05-21T17:19:55.205 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../095-python3-py_1.10.0-1_all.deb ... 2024-05-21T17:19:55.413 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-21T17:19:55.958 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pygments. 2024-05-21T17:19:55.987 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../096-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-21T17:19:56.025 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-21T17:19:56.252 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cachetools. 2024-05-21T17:19:56.279 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../070-python3-cachetools_5.0.0-1_all.deb ... 2024-05-21T17:19:56.319 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-21T17:19:56.428 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-toml. 2024-05-21T17:19:56.445 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../097-python3-toml_0.10.2-1_all.deb ... 2024-05-21T17:19:56.478 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-21T17:19:56.621 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-21T17:19:56.649 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-21T17:19:56.688 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-21T17:19:56.748 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pytest. 2024-05-21T17:19:56.776 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../098-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-21T17:19:56.814 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-21T17:19:57.009 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-21T17:19:57.036 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-21T17:19:57.067 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-21T17:19:57.160 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-simplejson. 2024-05-21T17:19:57.188 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../099-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-21T17:19:57.226 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-21T17:19:57.328 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rsa. 2024-05-21T17:19:57.355 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../073-python3-rsa_4.8-1_all.deb ... 2024-05-21T17:19:57.386 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-21T17:19:57.579 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-webob. 2024-05-21T17:19:57.607 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../100-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-21T17:19:57.645 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-21T17:19:57.665 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-google-auth. 2024-05-21T17:19:57.692 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-3_all.deb ... 2024-05-21T17:19:57.731 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-21T17:19:57.931 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-21T17:19:57.959 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../101-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-21T17:19:58.006 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-21T17:19:58.027 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-21T17:19:58.053 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../075-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-21T17:19:58.118 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-21T17:19:58.429 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-websocket. 2024-05-21T17:19:58.456 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../076-python3-websocket_1.2.3-1_all.deb ... 2024-05-21T17:19:58.496 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-21T17:19:58.685 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package radosgw. 2024-05-21T17:19:58.713 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../102-radosgw_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:58.751 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking radosgw (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:58.824 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-21T17:19:58.851 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../077-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-21T17:19:58.891 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-21T17:19:59.496 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-21T17:19:59.510 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../078-ceph-mgr-k8sevents_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:19:59.554 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:59.662 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package rbd-fuse. 2024-05-21T17:19:59.679 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../103-rbd-fuse_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:19:59.720 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking rbd-fuse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:19:59.874 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-21T17:19:59.902 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../079-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-21T17:19:59.932 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-21T17:20:00.008 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-21T17:20:00.267 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-21T17:20:00.286 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-21T17:20:00.314 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../080-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-21T17:20:00.352 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-21T17:20:00.368 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-21T17:20:00.630 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package jq. 2024-05-21T17:20:00.634 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-21T17:20:00.657 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../081-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-21T17:20:00.696 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-21T17:20:00.932 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package socat. 2024-05-21T17:20:00.948 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../082-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-21T17:20:01.012 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-21T17:20:01.101 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-21T17:20:01.413 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package xmlstarlet. 2024-05-21T17:20:01.441 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../083-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-21T17:20:01.453 INFO:teuthology.orchestra.run.smithi003.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-21T17:20:01.480 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-21T17:20:01.528 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up cephadm (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:01.843 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-test. 2024-05-21T17:20:01.859 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../084-ceph-test_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:20:01.925 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-test (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:02.053 INFO:teuthology.orchestra.run.smithi003.stdout:Adding system user cephadm....done 2024-05-21T17:20:02.201 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-21T17:20:02.443 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-21T17:20:02.569 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-21T17:20:02.829 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-21T17:20:03.088 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-21T17:20:03.214 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-21T17:20:03.532 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-21T17:20:03.876 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-21T17:20:04.119 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-21T17:20:04.412 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-21T17:20:04.465 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-volume. 2024-05-21T17:20:04.492 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../085-ceph-volume_19.0.0-3827-g99470715-1jammy_all.deb ... 2024-05-21T17:20:04.530 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-volume (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:04.663 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-21T17:20:04.851 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-21T17:20:04.868 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../086-libcephfs-dev_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:20:04.910 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs-dev (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:04.922 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-21T17:20:05.190 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-ceph-argparse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:05.238 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-21T17:20:05.266 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../087-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-21T17:20:05.304 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-21T17:20:05.441 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-21T17:20:05.550 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-21T17:20:05.602 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-21T17:20:05.629 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../088-lua-sec_1.0.2-1_amd64.deb ... 2024-05-21T17:20:05.668 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-21T17:20:05.676 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-21T17:20:05.936 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-21T17:20:06.064 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package nvme-cli. 2024-05-21T17:20:06.092 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../089-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-21T17:20:06.131 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-21T17:20:06.195 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-21T17:20:06.530 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package pkg-config. 2024-05-21T17:20:06.555 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-21T17:20:06.558 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../090-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-21T17:20:06.576 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-21T17:20:06.588 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-21T17:20:06.632 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-21T17:20:06.733 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-21T17:20:06.833 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-21T17:20:06.861 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../091-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-21T17:20:06.899 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-21T17:20:07.092 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-21T17:20:07.218 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-21T17:20:07.345 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-21T17:20:07.373 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../092-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-21T17:20:07.411 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-21T17:20:07.506 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-21T17:20:07.714 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pluggy. 2024-05-21T17:20:07.742 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../093-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-21T17:20:07.770 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-21T17:20:07.780 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-21T17:20:08.092 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-psutil. 2024-05-21T17:20:08.096 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up zip (3.0-12build2) ... 2024-05-21T17:20:08.120 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../094-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-21T17:20:08.158 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-21T17:20:08.222 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-21T17:20:08.504 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-py. 2024-05-21T17:20:08.532 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../095-python3-py_1.10.0-1_all.deb ... 2024-05-21T17:20:08.578 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-21T17:20:08.899 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pygments. 2024-05-21T17:20:08.899 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-21T17:20:08.927 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../096-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-21T17:20:08.982 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-21T17:20:09.009 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-21T17:20:09.301 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-21T17:20:09.445 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-toml. 2024-05-21T17:20:09.473 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../097-python3-toml_0.10.2-1_all.deb ... 2024-05-21T17:20:09.511 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-21T17:20:09.686 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-21T17:20:09.814 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pytest. 2024-05-21T17:20:09.842 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../098-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-21T17:20:09.872 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-21T17:20:09.945 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-21T17:20:10.226 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplejson. 2024-05-21T17:20:10.246 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-21T17:20:10.254 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../099-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-21T17:20:10.292 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-21T17:20:10.506 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-21T17:20:10.607 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-ceph-common (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:10.621 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webob. 2024-05-21T17:20:10.649 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../100-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-21T17:20:10.687 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-21T17:20:10.973 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-21T17:20:10.989 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../101-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-21T17:20:11.031 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-21T17:20:11.125 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-05-21T17:20:11.451 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-21T17:20:11.786 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package radosgw. 2024-05-21T17:20:11.814 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../102-radosgw_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:20:11.852 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking radosgw (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:12.682 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-21T17:20:12.864 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package rbd-fuse. 2024-05-21T17:20:12.892 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../103-rbd-fuse_19.0.0-3827-g99470715-1jammy_amd64.deb ... 2024-05-21T17:20:12.925 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:20:12.930 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking rbd-fuse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:13.059 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-21T17:20:13.242 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-21T17:20:13.427 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-21T17:20:13.494 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-21T17:20:13.553 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-21T17:20:13.620 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-21T17:20:13.670 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-21T17:20:13.879 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-21T17:20:13.955 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-21T17:20:14.184 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-21T17:20:14.223 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-21T17:20:14.466 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:20:14.575 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-21T17:20:14.589 INFO:teuthology.orchestra.run.smithi203.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-21T17:20:14.664 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up cephadm (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:14.893 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-21T17:20:15.114 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user cephadm....done 2024-05-21T17:20:15.152 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-21T17:20:15.254 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-21T17:20:15.420 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-21T17:20:15.505 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-21T17:20:15.546 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-21T17:20:15.606 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-21T17:20:15.830 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-21T17:20:15.832 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-21T17:20:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-21T17:20:16.083 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-21T17:20:16.209 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-21T17:20:16.250 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-21T17:20:16.500 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-21T17:20:16.511 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-21T17:20:16.626 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-21T17:20:16.752 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-21T17:20:16.841 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-21T17:20:16.871 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-21T17:20:17.096 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-21T17:20:17.106 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-21T17:20:17.222 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-21T17:20:17.365 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-21T17:20:17.609 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-21T17:20:17.632 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-21T17:20:17.733 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-21T17:20:17.860 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-21T17:20:17.967 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:20:18.093 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-21T17:20:18.120 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-argparse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:18.379 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-21T17:20:18.419 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-21T17:20:18.480 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-21T17:20:18.720 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-21T17:20:18.823 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-21T17:20:18.996 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-21T17:20:19.007 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-21T17:20:19.097 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-21T17:20:19.373 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-21T17:20:19.375 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-21T17:20:19.616 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-21T17:20:19.760 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-21T17:20:19.789 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-21T17:20:19.854 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-21T17:20:19.980 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-21T17:20:19.984 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-21T17:20:20.110 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librados2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:20.236 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:20.354 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-21T17:20:20.356 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-21T17:20:20.473 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-21T17:20:20.648 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libcephfs2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:20.733 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-21T17:20:20.774 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libradosstriper1 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:20.950 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-21T17:20:21.018 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-21T17:20:21.226 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librbd1 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:21.346 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up zip (3.0-12build2) ... 2024-05-21T17:20:21.352 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-modules-core (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:21.470 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-21T17:20:21.478 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-fuse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:21.775 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-21T17:20:21.775 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-21T17:20:22.147 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-21T17:20:22.257 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-21T17:20:22.290 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libcephfs-dev (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:22.433 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rados (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:22.549 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-21T17:20:22.559 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librgw2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:22.685 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-21T17:20:22.934 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-21T17:20:23.186 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-21T17:20:23.487 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-21T17:20:23.738 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-21T17:20:23.864 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-common (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:23.897 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rbd (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:24.032 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up rbd-fuse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:24.158 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rgw (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:24.190 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-05-21T17:20:24.284 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cephfs (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:24.410 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-common (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:24.550 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-21T17:20:24.961 INFO:teuthology.orchestra.run.smithi003.stdout:Adding group ceph....done 2024-05-21T17:20:25.338 INFO:teuthology.orchestra.run.smithi003.stdout:Adding system user ceph....done 2024-05-21T17:20:25.506 INFO:teuthology.orchestra.run.smithi003.stdout:Setting system user ceph properties....done 2024-05-21T17:20:25.523 INFO:teuthology.orchestra.run.smithi003.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-21T17:20:25.669 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-21T17:20:25.781 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-21T17:20:26.025 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:20:26.043 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-21T17:20:26.134 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-21T17:20:26.385 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-21T17:20:26.502 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-test (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:26.511 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-21T17:20:26.628 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up radosgw (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:26.638 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-21T17:20:26.931 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-21T17:20:27.087 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-21T17:20:27.087 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-21T17:20:27.182 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-21T17:20:27.416 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:20:27.526 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-21T17:20:27.555 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-base (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:27.828 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-21T17:20:27.908 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-21T17:20:28.078 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-21T17:20:28.347 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-21T17:20:28.390 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mds (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:28.456 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-21T17:20:28.618 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-21T17:20:28.618 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-21T17:20:28.724 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-21T17:20:28.867 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-21T17:20:29.084 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:29.176 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-21T17:20:29.309 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-21T17:20:29.309 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-21T17:20:29.436 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-21T17:20:29.562 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-21T17:20:29.688 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-21T17:20:29.777 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-21T17:20:29.818 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-osd (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:30.006 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-21T17:20:30.095 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-21T17:20:30.095 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-21T17:20:30.107 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-21T17:20:30.501 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-21T17:20:30.612 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:30.627 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-21T17:20:30.738 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:30.878 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-21T17:20:30.914 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mon (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:31.004 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-21T17:20:31.142 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-21T17:20:31.142 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-21T17:20:31.322 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-21T17:20:31.615 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-21T17:20:31.702 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-cephadm (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:31.853 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:31.874 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-21T17:20:31.962 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-dashboard (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:32.000 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-21T17:20:32.105 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-volume (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:32.285 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-21T17:20:32.474 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-21T17:20:32.545 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-21T17:20:32.584 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-21T17:20:32.728 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-21T17:20:32.963 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-21T17:20:33.081 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librados2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:33.190 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:33.291 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-21T17:20:33.628 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:33.754 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libradosstriper1 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:33.878 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-21T17:20:33.879 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-21T17:20:34.189 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librbd1 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:34.524 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-modules-core (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:34.540 INFO:teuthology.orchestra.run.smithi003.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-21T17:20:34.546 DEBUG:teuthology.parallel:result is None 2024-05-21T17:20:34.633 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-fuse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:34.931 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-21T17:20:34.932 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-21T17:20:35.370 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs-dev (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:35.497 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rados (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:35.614 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librgw2 (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:35.724 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-21T17:20:36.914 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rbd (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:37.038 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up rbd-fuse (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:37.164 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rgw (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:37.290 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cephfs (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:37.408 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-common (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:37.918 INFO:teuthology.orchestra.run.smithi203.stdout:Adding group ceph....done 2024-05-21T17:20:38.211 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user ceph....done 2024-05-21T17:20:38.380 INFO:teuthology.orchestra.run.smithi203.stdout:Setting system user ceph properties....done 2024-05-21T17:20:38.398 INFO:teuthology.orchestra.run.smithi203.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-21T17:20:38.534 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-21T17:20:38.880 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-21T17:20:39.327 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-test (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:39.453 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up radosgw (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:40.089 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-21T17:20:40.089 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-21T17:20:40.463 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-base (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:40.807 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-21T17:20:41.257 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mds (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:41.465 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-21T17:20:41.465 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-21T17:20:41.942 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:42.172 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-21T17:20:42.172 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-21T17:20:42.644 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-osd (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:42.921 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-21T17:20:42.921 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-21T17:20:43.387 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:43.513 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:43.639 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mon (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:43.835 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-21T17:20:43.836 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-21T17:20:44.324 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-cephadm (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:44.458 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:44.585 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-dashboard (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:44.744 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-volume (19.0.0-3827-g99470715-1jammy) ... 2024-05-21T17:20:45.097 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-21T17:20:45.265 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-21T17:20:45.518 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-21T17:20:46.552 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-21T17:20:47.198 INFO:teuthology.orchestra.run.smithi203.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-21T17:20:47.203 DEBUG:teuthology.parallel:result is None 2024-05-21T17:20:47.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=99470715356be3e23220ec4a5eacc2bc4dfde33f 2024-05-21T17:20:47.345 DEBUG:teuthology.orchestra.run.smithi003:> dpkg-query -W -f '${Version}' ceph 2024-05-21T17:20:47.365 INFO:teuthology.orchestra.run.smithi003.stdout:19.0.0-3827-g99470715-1jammy 2024-05-21T17:20:47.365 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3827-g99470715-1jammy 2024-05-21T17:20:47.366 INFO:teuthology.task.install:The correct ceph version 19.0.0-3827-g99470715-1jammy is installed. 2024-05-21T17:20:47.367 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=99470715356be3e23220ec4a5eacc2bc4dfde33f 2024-05-21T17:20:47.575 DEBUG:teuthology.orchestra.run.smithi203:> dpkg-query -W -f '${Version}' ceph 2024-05-21T17:20:47.595 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-3827-g99470715-1jammy 2024-05-21T17:20:47.595 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3827-g99470715-1jammy 2024-05-21T17:20:47.595 INFO:teuthology.task.install:The correct ceph version 19.0.0-3827-g99470715-1jammy is installed. 2024-05-21T17:20:47.597 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-21T17:20:47.597 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:47.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-21T17:20:47.612 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:20:47.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-21T17:20:47.652 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-21T17:20:47.653 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:47.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-05-21T17:20:47.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-21T17:20:47.724 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:20:47.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-05-21T17:20:47.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-21T17:20:47.790 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-21T17:20:47.791 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:47.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-21T17:20:47.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-21T17:20:47.859 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:20:47.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-21T17:20:47.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-21T17:20:47.926 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-21T17:20:47.927 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:47.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-05-21T17:20:47.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-21T17:20:47.996 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:20:47.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-05-21T17:20:48.009 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-21T17:20:48.064 INFO:teuthology.run_tasks:Running task ceph... 2024-05-21T17:20:48.141 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-21T17:20:48.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2024-05-21T17:20:48.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-05-21T17:20:48.156 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-21T17:20:48.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-21T17:20:48.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-21T17:20:48.211 INFO:tasks.ceph:Creating extra log directories... 2024-05-21T17:20:48.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-21T17:20:48.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-21T17:20:48.276 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-21T17:20:48.276 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-05-21T17:20:48.277 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922', 'branch': 'wip-yuri3-testing-2024-05-20-1050', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7716922', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2024-05-20-1050'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'mon_bind_msgr2': False, 'sha1': '99470715356be3e23220ec4a5eacc2bc4dfde33f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '99470715356be3e23220ec4a5eacc2bc4dfde33f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2024-05-20-1050', 'sha1': '99470715356be3e23220ec4a5eacc2bc4dfde33f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9006, 'sha1': '99470715356be3e23220ec4a5eacc2bc4dfde33f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri3-testing-2024-05-20-1050', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_99470715356be3e23220ec4a5eacc2bc4dfde33f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '99470715356be3e23220ec4a5eacc2bc4dfde33f', 'targets': {'smithi003.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEDkmWKUi7K3N7gP72K0zGzCXHsXvDsIwLfpqskkN5nvG8JMm5kR2o89H+tQf+W9Jv2nlWfgMOzUzi0mO843moY=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIe6uKHa3bwWj2QHbMHGqluB9B9BIwqLpzyQ5jtvs+rxt7EDj/vJt75l0VWO57i4nBVWrd6K5TkeVxQkmdWWK3U='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a', 'timestamp': '2024-05-20_20:45:46', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-05-21T17:20:48.277 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-21T17:20:48.309 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-21T17:20:48.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2024-05-21T17:20:48.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-05-21T17:20:48.372 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:48.372 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-05-21T17:20:48.421 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-21T17:20:48.421 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-05-21T17:20:48.469 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-21T17:20:48.469 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T17:20:48.469 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 868 Links: 1 2024-05-21T17:20:48.469 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T17:20:48.469 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-05-21 17:15:31.299173202 +0000 2024-05-21T17:20:48.469 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-05-21 17:15:31.023175667 +0000 2024-05-21T17:20:48.469 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-05-21 17:15:31.023175667 +0000 2024-05-21T17:20:48.469 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-05-21T17:20:48.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-21T17:20:48.523 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-05-21T17:20:48.524 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-05-21T17:20:48.524 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.0004724 s, 1.1 MB/s 2024-05-21T17:20:48.525 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-21T17:20:48.573 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-05-21T17:20:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-21T17:20:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T17:20:48.620 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-05-21T17:20:48.620 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T17:20:48.620 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-05-21 17:15:31.787168844 +0000 2024-05-21T17:20:48.621 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-05-21 17:15:31.531171130 +0000 2024-05-21T17:20:48.621 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-05-21 17:15:31.531171130 +0000 2024-05-21T17:20:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-05-21T17:20:48.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-21T17:20:48.675 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-05-21T17:20:48.675 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-05-21T17:20:48.675 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000442462 s, 1.2 MB/s 2024-05-21T17:20:48.676 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-21T17:20:48.725 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-05-21T17:20:48.773 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-21T17:20:48.773 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T17:20:48.773 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 901 Links: 1 2024-05-21T17:20:48.773 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T17:20:48.773 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-05-21 17:15:32.287164378 +0000 2024-05-21T17:20:48.773 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-05-21 17:15:32.027166700 +0000 2024-05-21T17:20:48.773 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-05-21 17:15:32.027166700 +0000 2024-05-21T17:20:48.773 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-05-21T17:20:48.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-21T17:20:48.827 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-05-21T17:20:48.827 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-05-21T17:20:48.827 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000349302 s, 1.5 MB/s 2024-05-21T17:20:48.828 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-21T17:20:48.877 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-05-21T17:20:48.924 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-21T17:20:48.924 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T17:20:48.925 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-05-21T17:20:48.925 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T17:20:48.925 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-05-21 17:15:32.539162126 +0000 2024-05-21T17:20:48.925 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-05-21 17:15:32.535162162 +0000 2024-05-21T17:20:48.925 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-05-21 17:15:32.535162162 +0000 2024-05-21T17:20:48.925 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-05-21T17:20:48.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-21T17:20:48.979 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-05-21T17:20:48.979 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-05-21T17:20:48.979 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000417877 s, 1.2 MB/s 2024-05-21T17:20:48.981 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-21T17:20:49.029 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-05-21T17:20:49.030 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:20:49.030 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-05-21T17:20:49.036 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-21T17:20:49.036 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-05-21T17:20:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-21T17:20:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T17:20:49.084 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 869 Links: 1 2024-05-21T17:20:49.085 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T17:20:49.085 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-05-21 17:15:29.065898643 +0000 2024-05-21T17:20:49.085 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-05-21 17:15:29.061898679 +0000 2024-05-21T17:20:49.085 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-05-21 17:15:29.061898679 +0000 2024-05-21T17:20:49.085 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-05-21T17:20:49.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-21T17:20:49.139 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-05-21T17:20:49.139 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-05-21T17:20:49.139 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.00045213 s, 1.1 MB/s 2024-05-21T17:20:49.140 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-21T17:20:49.189 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-05-21T17:20:49.236 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-21T17:20:49.236 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T17:20:49.236 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-05-21T17:20:49.236 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T17:20:49.236 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-05-21 17:15:29.785892220 +0000 2024-05-21T17:20:49.236 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-05-21 17:15:29.513894647 +0000 2024-05-21T17:20:49.236 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-05-21 17:15:29.513894647 +0000 2024-05-21T17:20:49.236 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-05-21T17:20:49.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-21T17:20:49.289 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-05-21T17:20:49.290 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-05-21T17:20:49.290 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000312942 s, 1.6 MB/s 2024-05-21T17:20:49.291 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-21T17:20:49.337 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-05-21T17:20:49.384 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-21T17:20:49.384 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T17:20:49.384 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-05-21T17:20:49.384 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T17:20:49.384 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-05-21 17:15:30.265887938 +0000 2024-05-21T17:20:49.384 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-05-21 17:15:30.013890186 +0000 2024-05-21T17:20:49.384 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-05-21 17:15:30.013890186 +0000 2024-05-21T17:20:49.384 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-05-21T17:20:49.384 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-21T17:20:49.438 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-05-21T17:20:49.438 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-05-21T17:20:49.438 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000401828 s, 1.3 MB/s 2024-05-21T17:20:49.439 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-21T17:20:49.489 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-05-21T17:20:49.536 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-21T17:20:49.536 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T17:20:49.536 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-05-21T17:20:49.536 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T17:20:49.536 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-05-21 17:15:30.753883585 +0000 2024-05-21T17:20:49.536 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-05-21 17:15:30.513885726 +0000 2024-05-21T17:20:49.536 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-05-21 17:15:30.513885726 +0000 2024-05-21T17:20:49.536 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-05-21T17:20:49.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-21T17:20:49.589 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-05-21T17:20:49.589 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-05-21T17:20:49.590 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000321933 s, 1.6 MB/s 2024-05-21T17:20:49.591 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-21T17:20:49.637 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-05-21T17:20:49.637 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-05-21T17:20:49.637 INFO:tasks.ceph:Generating config... 2024-05-21T17:20:49.638 INFO:tasks.ceph:[client] debug monc = 20 2024-05-21T17:20:49.638 INFO:tasks.ceph:[client] debug ms = 1 2024-05-21T17:20:49.638 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-21T17:20:49.639 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-05-21T17:20:49.639 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-05-21T17:20:49.639 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-21T17:20:49.639 INFO:tasks.ceph:[global] ms type = async 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-05-21T17:20:49.639 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-05-21T17:20:49.640 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-05-21T17:20:49.640 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-05-21T17:20:49.640 INFO:tasks.ceph:[mon] paxos min = 10 2024-05-21T17:20:49.640 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-05-21T17:20:49.640 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-21T17:20:49.640 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-05-21T17:20:49.641 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-05-21T17:20:49.641 INFO:tasks.ceph:Setting up mon.a... 2024-05-21T17:20:49.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-21T17:20:49.677 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2024-05-21T17:20:49.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-21T17:20:49.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-21T17:20:49.824 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.3:6789]'), ('mon.b', '[v1:172.21.15.3:6790]'), ('mon.c', '[v1:172.21.15.3:6791]'), ('mon.d', '[v1:172.21.15.3:6792]'), ('mon.e', '[v1:172.21.15.3:6793]'), ('mon.f', '[v1:172.21.15.203:6789]'), ('mon.g', '[v1:172.21.15.203:6790]'), ('mon.h', '[v1:172.21.15.203:6791]'), ('mon.i', '[v1:172.21.15.203:6792]')] 2024-05-21T17:20:49.824 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.3:6791],[v1:172.21.15.3:6792],[v1:172.21.15.3:6793],[v1:172.21.15.203:6789],[v1:172.21.15.203:6790],[v1:172.21.15.203:6791],[v1:172.21.15.203:6792]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-05-21T17:20:49.825 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:49.825 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-21T17:20:49.872 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.3:6789]' --addv b '[v1:172.21.15.3:6790]' --addv c '[v1:172.21.15.3:6791]' --addv d '[v1:172.21.15.3:6792]' --addv e '[v1:172.21.15.3:6793]' --addv f '[v1:172.21.15.203:6789]' --addv g '[v1:172.21.15.203:6790]' --addv h '[v1:172.21.15.203:6791]' --addv i '[v1:172.21.15.203:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = quincy 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 17 (quincy) 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:election_strategy: 3 2024-05-21T17:20:49.979 INFO:teuthology.orchestra.run.smithi003.stdout:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:1: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:2: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:3: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:4: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:5: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:6: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:7: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:8: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:20:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-05-21T17:20:49.981 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-21T17:20:49.986 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4db3b12d-01c4-4d6c-b29b-1648ee37f87f... 2024-05-21T17:20:49.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-21T17:20:50.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-21T17:20:50.057 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-05-21T17:20:50.057 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2024-05-21T17:20:50.057 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-21T17:20:50.057 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2024-05-21T17:20:50.057 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.057 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2024-05-21T17:20:50.057 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.057 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-21T17:20:50.058 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-21T17:20:50.059 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.3:6791],[v1:172.21.15.3:6792],[v1:172.21.15.3:6793],[v1:172.21.15.203:6789],[v1:172.21.15.203:6790],[v1:172.21.15.203:6791],[v1:172.21.15.203:6792]" 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: mon client directed command retry = 5 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: mon election default strategy = 3 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: ms bind msgr2 = False 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject socket failures = 5000 2024-05-21T17:20:50.060 INFO:teuthology.orchestra.run.smithi003.stdout: ms type = async 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2024-05-21T17:20:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug trim objects = true 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.062 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression algorithm = lz4 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression mode = aggressive 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore zero block detection = True 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 20 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 20 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2024-05-21T17:20:50.063 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 10 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify cached snaps = True 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify missing on start = True 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock override recovery settings = True 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue = debug_random 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue cut off = debug_random 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-21T17:20:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2024-05-21T17:20:50.065 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.066 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon min osdmap epochs = 25 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune interval = 2 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune min = 15 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune txsize = 2 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon scrub inject crc mismatch = 0.01 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon scrub inject missing keys = 0.05 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: mon scrub interval = 300 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: paxos min = 10 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: paxos service trim min = 5 2024-05-21T17:20:50.067 INFO:teuthology.orchestra.run.smithi003.stdout: paxos trim min = 10 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.d] 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.e] 2024-05-21T17:20:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.f] 2024-05-21T17:20:50.069 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.g] 2024-05-21T17:20:50.069 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.h] 2024-05-21T17:20:50.069 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.i] 2024-05-21T17:20:50.069 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-05-21T17:20:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-05-21T17:20:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-21T17:20:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-05-21T17:20:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-05-21T17:20:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-21T17:20:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.3:6791],[v1:172.21.15.3:6792],[v1:172.21.15.3:6793],[v1:172.21.15.203:6789],[v1:172.21.15.203:6790],[v1:172.21.15.203:6791],[v1:172.21.15.203:6792]" 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: mon client directed command retry = 5 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 3 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: ms bind msgr2 = False 2024-05-21T17:20:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject socket failures = 5000 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: ms type = async 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-05-21T17:20:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-05-21T17:20:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore compression algorithm = lz4 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore compression mode = aggressive 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 20 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 20 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 10 2024-05-21T17:20:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-05-21T17:20:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-05-21T17:20:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 25 2024-05-21T17:20:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub inject crc mismatch = 0.01 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub inject missing keys = 0.05 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: paxos min = 10 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 5 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: paxos trim min = 10 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-21T17:20:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.d] 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.e] 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.f] 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.g] 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.h] 2024-05-21T17:20:50.080 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.i] 2024-05-21T17:20:50.081 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-21T17:20:50.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-21T17:20:50.175 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-21T17:20:50.175 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:50.175 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-21T17:20:50.180 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:50.180 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-21T17:20:50.228 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:20:50.228 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:50.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-21T17:20:50.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-21T17:20:50.292 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:20:50.292 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:20:50.340 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:20:50.340 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:20:50.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-21T17:20:50.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-21T17:20:50.364 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:20:50.364 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:20:50.416 INFO:tasks.ceph:Setting up mon nodes... 2024-05-21T17:20:50.416 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-21T17:20:50.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-21T17:20:50.501 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-21T17:20:50.517 INFO:tasks.ceph:Setting up mds nodes... 2024-05-21T17:20:50.517 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-21T17:20:50.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-21T17:20:50.592 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-21T17:20:50.603 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-21T17:20:50.603 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-05-21T17:20:50.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-21T17:20:50.618 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-05-21T17:20:50.618 INFO:tasks.ceph:role: osd.0 2024-05-21T17:20:50.618 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:20:50.618 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-21T17:20:50.680 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-21T17:20:51.142 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-05-21T17:20:51.144 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-05-21T17:20:51.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-21T17:20:51.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-21T17:20:51.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2024-05-21T17:20:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:20:51.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-21T17:20:51.372 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-05-21T17:20:51.372 INFO:tasks.ceph:role: osd.1 2024-05-21T17:20:51.372 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:20:51.372 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-21T17:20:51.436 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-21T17:20:51.876 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-05-21T17:20:51.878 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-05-21T17:20:51.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-21T17:20:51.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-21T17:20:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:20:51.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2024-05-21T17:20:51.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-21T17:20:52.036 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-05-21T17:20:52.036 INFO:tasks.ceph:role: osd.2 2024-05-21T17:20:52.036 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:20:52.037 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-21T17:20:52.099 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-21T17:20:52.099 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-21T17:20:52.099 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-21T17:20:52.099 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-21T17:20:52.100 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-21T17:20:52.100 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-05-21T17:20:52.100 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-21T17:20:52.100 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-21T17:20:52.100 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-21T17:20:52.100 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-21T17:20:52.562 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-05-21T17:20:52.564 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-05-21T17:20:52.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-21T17:20:52.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-21T17:20:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:20:52.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2024-05-21T17:20:52.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:20:52.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:52.796+0000 7f69b730a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-21T17:20:52.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:52.796+0000 7f69b730a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-21T17:20:52.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:52.796+0000 7f69b730a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-21T17:20:52.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:52.796+0000 7f69b730a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-21T17:20:52.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:52.796+0000 7f69b730a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-21T17:20:52.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:52.804+0000 7f69b730a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-21T17:20:55.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-21T17:20:55.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:20:55.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:55.880+0000 7fe7655267c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-21T17:20:55.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:55.884+0000 7fe7655267c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-21T17:20:55.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:55.884+0000 7fe7655267c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-21T17:20:55.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:55.884+0000 7fe7655267c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-21T17:20:55.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:55.884+0000 7fe7655267c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-21T17:20:55.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:55.884+0000 7fe7655267c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-21T17:20:58.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-21T17:20:58.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:20:59.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:59.012+0000 7f5cd397c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-21T17:20:59.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:59.016+0000 7f5cd397c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-21T17:20:59.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:59.016+0000 7f5cd397c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-21T17:20:59.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:59.016+0000 7f5cd397c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-21T17:20:59.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:59.016+0000 7f5cd397c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-21T17:20:59.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:20:59.016+0000 7f5cd397c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-21T17:21:01.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-21T17:21:02.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-21T17:21:02.056 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-05-21T17:21:02.056 INFO:tasks.ceph:role: osd.3 2024-05-21T17:21:02.056 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:21:02.057 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-21T17:21:02.120 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-21T17:21:02.121 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-21T17:21:02.572 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-05-21T17:21:02.574 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-05-21T17:21:02.574 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-05-21T17:21:02.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-21T17:21:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:21:02.679 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-05-21T17:21:02.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-21T17:21:02.735 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-05-21T17:21:02.736 INFO:tasks.ceph:role: osd.4 2024-05-21T17:21:02.736 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:21:02.736 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-21T17:21:02.799 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-21T17:21:03.260 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-05-21T17:21:03.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-05-21T17:21:03.263 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-05-21T17:21:03.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-21T17:21:03.426 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-05-21T17:21:03.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:21:03.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-21T17:21:03.487 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-05-21T17:21:03.488 INFO:tasks.ceph:role: osd.5 2024-05-21T17:21:03.488 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:21:03.488 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-21T17:21:03.551 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-21T17:21:03.552 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-21T17:21:04.004 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-05-21T17:21:04.007 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-05-21T17:21:04.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-05-21T17:21:04.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-21T17:21:04.157 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-05-21T17:21:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:21:04.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:21:04.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:04.286+0000 7fc98d9807c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-21T17:21:04.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:04.286+0000 7fc98d9807c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-21T17:21:04.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:04.286+0000 7fc98d9807c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-21T17:21:04.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:04.286+0000 7fc98d9807c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-21T17:21:04.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:04.286+0000 7fc98d9807c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-21T17:21:04.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:04.290+0000 7fc98d9807c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-21T17:21:07.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-21T17:21:07.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:21:07.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:07.354+0000 7fe88d5417c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-21T17:21:07.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:07.354+0000 7fe88d5417c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-21T17:21:07.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:07.354+0000 7fe88d5417c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-21T17:21:07.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:07.354+0000 7fe88d5417c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-21T17:21:07.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:07.354+0000 7fe88d5417c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-21T17:21:07.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:07.354+0000 7fe88d5417c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-21T17:21:10.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-21T17:21:10.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:21:10.490 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:10.490+0000 7f170f3997c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-21T17:21:10.490 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:10.490+0000 7f170f3997c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-21T17:21:10.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:10.490+0000 7f170f3997c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-21T17:21:10.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:10.490+0000 7f170f3997c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-21T17:21:10.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:10.490+0000 7f170f3997c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-21T17:21:10.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:10.494+0000 7f170f3997c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-21T17:21:13.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-21T17:21:13.528 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-21T17:21:13.528 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:21:13.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-21T17:21:13.543 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:21:13.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-21T17:21:13.601 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:21:13.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-21T17:21:13.660 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:21:13.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-21T17:21:13.673 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:21:13.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-21T17:21:13.727 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:21:13.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-21T17:21:13.783 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:21:13.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-21T17:21:13.839 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:21:13.839 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-21T17:21:13.888 INFO:tasks.ceph:Adding keys to all mons... 2024-05-21T17:21:13.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-21T17:21:13.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-21T17:21:13.937 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2024-05-21T17:21:13.937 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQD010xmdK3LLxAASzoZ5GXL4n2Ch8FRzzIilg== 2024-05-21T17:21:13.937 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2024-05-21T17:21:13.937 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQD310xmwuXqNBAAlO94cyktcb/fBH1gxzR04g== 2024-05-21T17:21:13.937 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2024-05-21T17:21:13.937 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQD710xmlEsyARAAbHW3B/dfiFJqbL4n3UUHzg== 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDy10xmnwkYHhAAIqOq6IQ2fo2FnsIB+9fSXw== 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.3] 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAA2ExmvJJMERAAUff2mD0vnbSVm2f5YTCTGg== 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.4] 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAD2ExmwsRNFRAArHpitjbPNJBCHga+aIZRaw== 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.5] 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAG2ExmM1ltHRAAaz/Hro1rTknyPp20tY3V1g== 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-05-21T17:21:13.938 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDy10xmzZSMIxAAi6OrL280rSVAGJOseAtKjw== 2024-05-21T17:21:13.945 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-05-21T17:21:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD010xmdK3LLxAASzoZ5GXL4n2Ch8FRzzIilg== 2024-05-21T17:21:13.945 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD310xmwuXqNBAAlO94cyktcb/fBH1gxzR04g== 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD710xmlEsyARAAbHW3B/dfiFJqbL4n3UUHzg== 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDy10xmnwkYHhAAIqOq6IQ2fo2FnsIB+9fSXw== 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAA2ExmvJJMERAAUff2mD0vnbSVm2f5YTCTGg== 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAD2ExmwsRNFRAArHpitjbPNJBCHga+aIZRaw== 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAG2ExmM1ltHRAAaz/Hro1rTknyPp20tY3V1g== 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-05-21T17:21:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDy10xmzZSMIxAAi6OrL280rSVAGJOseAtKjw== 2024-05-21T17:21:13.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:13.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-21T17:21:14.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-21T17:21:14.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-21T17:21:14.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-21T17:21:14.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-21T17:21:14.594 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-21T17:21:14.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-21T17:21:14.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:14.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-21T17:21:14.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-21T17:21:14.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:14.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-21T17:21:14.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-21T17:21:14.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:15.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-21T17:21:15.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-05-21T17:21:15.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:15.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-05-21T17:21:15.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-05-21T17:21:15.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:15.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-05-21T17:21:15.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-05-21T17:21:15.360 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:15.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-05-21T17:21:15.469 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-05-21T17:21:15.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:15.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-05-21T17:21:15.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-05-21T17:21:15.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:15.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-05-21T17:21:15.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-05-21T17:21:15.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-21T17:21:15.917 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-05-21T17:21:15.929 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:21:15.931 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-21T17:21:15.975 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-21T17:21:15.975 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-21T17:21:15.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-21T17:21:15.977 INFO:tasks.ceph.mon.a:Started 2024-05-21T17:21:15.978 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-21T17:21:15.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-21T17:21:15.980 INFO:tasks.ceph.mon.b:Started 2024-05-21T17:21:15.980 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-21T17:21:15.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-21T17:21:15.982 INFO:tasks.ceph.mon.c:Started 2024-05-21T17:21:15.983 INFO:tasks.ceph.mon.d:Restarting daemon 2024-05-21T17:21:15.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-05-21T17:21:15.985 INFO:tasks.ceph.mon.d:Started 2024-05-21T17:21:15.985 INFO:tasks.ceph.mon.e:Restarting daemon 2024-05-21T17:21:15.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-05-21T17:21:15.987 INFO:tasks.ceph.mon.e:Started 2024-05-21T17:21:15.988 INFO:tasks.ceph.mon.f:Restarting daemon 2024-05-21T17:21:15.988 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-05-21T17:21:16.021 INFO:tasks.ceph.mon.f:Started 2024-05-21T17:21:16.021 INFO:tasks.ceph.mon.g:Restarting daemon 2024-05-21T17:21:16.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-05-21T17:21:16.023 INFO:tasks.ceph.mon.g:Started 2024-05-21T17:21:16.023 INFO:tasks.ceph.mon.h:Restarting daemon 2024-05-21T17:21:16.023 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-05-21T17:21:16.025 INFO:tasks.ceph.mon.h:Started 2024-05-21T17:21:16.025 INFO:tasks.ceph.mon.i:Restarting daemon 2024-05-21T17:21:16.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-05-21T17:21:16.027 INFO:tasks.ceph.mon.i:Started 2024-05-21T17:21:16.027 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-21T17:21:16.028 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-21T17:21:16.028 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-21T17:21:16.030 INFO:tasks.ceph.mgr.x:Started 2024-05-21T17:21:16.030 INFO:tasks.ceph:Setting crush tunables to default 2024-05-21T17:21:16.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient: get_monmap_and_config 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient: build_initial_monmap 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient: monmap: 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:16.162137+0000 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:16.162137+0000 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:16.163 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 1 Processor -- start 2024-05-21T17:21:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 1 -- start start 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient: init 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient: _add_conns ranks=[4,8,5,6,3,1,0,2,7] 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient(hunting): picked mon.noname-c con 0x7f34b40a5bd0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient(hunting): picked mon.noname-e con 0x7f34b40a66c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient(hunting): picked mon.noname-h con 0x7f34b409db30 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b409d430 con 0x7f34b40a5bd0 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b40a7680 con 0x7f34b409db30 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b40a6d00 con 0x7f34b40a66c0 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34bc381640 10 monclient(hunting): authenticate will time out at 943.319300s 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34b991c640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f34b40a66c0 0x7f34b413dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:56932/0 (socket says 172.21.15.3:56932) 2024-05-21T17:21:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34b991c640 1 -- 172.21.15.3:0/1236841183 learned_addr learned my addr 172.21.15.3:0/1236841183 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:16.156+0000 7f34ba11d640 1 --1- 172.21.15.3:0/1236841183 >> v1:172.21.15.3:6791/0 conn(0x7f34b40a5bd0 0x7f34b40a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:45724/0 (socket says 172.21.15.3:45724) 2024-05-21T17:21:19.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 10 monclient(hunting): tick 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 10 monclient(hunting): _check_auth_tickets 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 1 monclient(hunting): continuing hunt 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 10 monclient(hunting): _reopen_session rank -1 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.3:6793/0 conn(0x7f34b40a66c0 legacy=0x7f34b413dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.203:6791/0 conn(0x7f34b409db30 legacy=0x7f34b41415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.3:6791/0 conn(0x7f34b40a5bd0 legacy=0x7f34b40a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 10 monclient: _add_conns ranks=[2,6,1,3,7,0] 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-b con 0x7f34980037f0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-d con 0x7f3498007480 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-f con 0x7f349800b210 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b409df40 con 0x7f349800b210 2024-05-21T17:21:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3498010030 con 0x7f34980037f0 2024-05-21T17:21:19.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34980112b0 con 0x7f3498007480 2024-05-21T17:21:19.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:19.156+0000 7f34abfff640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 10 monclient(hunting): tick 2024-05-21T17:21:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 10 monclient(hunting): _check_auth_tickets 2024-05-21T17:21:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 1 monclient(hunting): continuing hunt 2024-05-21T17:21:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 10 monclient(hunting): _reopen_session rank -1 2024-05-21T17:21:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.3:6792/0 conn(0x7f3498007480 legacy=0x7f3498009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.3:6790/0 conn(0x7f34980037f0 legacy=0x7f3498005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.203:6789/0 conn(0x7f349800b210 legacy=0x7f349800d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 10 monclient: _add_conns ranks=[0,7,3] 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-a con 0x7f3498011700 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34ba91e640 1 -- 172.21.15.3:0/1236841183 reap_dead start 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-i con 0x7f3498010200 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-g con 0x7f34980178f0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3498012590 con 0x7f3498011700 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.156+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f349801c410 con 0x7f34980178f0 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.160+0000 7f34ba91e640 1 -- 172.21.15.3:0/1236841183 reap_dead start 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.160+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3498006410 con 0x7f3498010200 2024-05-21T17:21:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:22.160+0000 7f34abfff640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:25.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 10 monclient(hunting): tick 2024-05-21T17:21:25.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 10 monclient(hunting): _check_auth_tickets 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 1 monclient(hunting): continuing hunt 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 10 monclient(hunting): _reopen_session rank -1 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.203:6792/0 conn(0x7f3498010200 legacy=0x7f34980106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.203:6790/0 conn(0x7f34980178f0 legacy=0x7f3498019ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.3:6789/0 conn(0x7f3498011700 legacy=0x7f3498011b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 10 monclient: _add_conns ranks=[5,7,4,2,1,3,0,8,6] 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-h con 0x7f3498006950 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-i con 0x7f3498006dd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 10 monclient(hunting): picked mon.noname-c con 0x7f349800a110 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3498006630 con 0x7f349800a110 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3498010630 con 0x7f3498006950 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3498010800 con 0x7f3498006dd0 2024-05-21T17:21:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:25.160+0000 7f34abfff640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:26.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394385103 0 0) 0x7f3498010800 con 0x7f3498006dd0 2024-05-21T17:21:26.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:26.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:26.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:26.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): my global_id is 4111 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34a0003530 con 0x7f3498006dd0 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3998791107 0 0) 0x7f3498010630 con 0x7f3498006950 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): my global_id is 4118 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3498010800 con 0x7f3498006950 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1535724520 0 0) 0x7f3498006630 con 0x7f349800a110 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:26.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient(hunting): my global_id is 4117 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3498010630 con 0x7f349800a110 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4130404460 0 0) 0x7f34a0003530 con 0x7f3498006dd0 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.203:6791/0 conn(0x7f3498006950 legacy=0x7f349800d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.3:6791/0 conn(0x7f349800a110 legacy=0x7f34980098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: _finish_hunting 0 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 monclient: found mon.noname-i 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.203:6792/0 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 --> v1:172.21.15.203:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3498010a70 con 0x7f3498006dd0 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: _finish_auth 0 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: _check_auth_tickets 2024-05-21T17:21:26.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f34a40017a0 con 0x7f3498006dd0 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 5 monclient: authenticate success, global_id 4111 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: dump: 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:26.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:26.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 <== mon.7 v1:172.21.15.203:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f34a4001940 con 0x7f3498006dd0 2024-05-21T17:21:26.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 1 -- 172.21.15.3:0/1236841183 <== mon.7 v1:172.21.15.203:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f34a4004bf0 con 0x7f3498006dd0 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34b911b640 10 monclient: dump: 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 4 set_mon_vals no callback set 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 10 monclient: shutdown 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 1 -- 172.21.15.3:0/1236841183 >> v1:172.21.15.203:6792/0 conn(0x7f3498006dd0 legacy=0x7f3498005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34ba91e640 1 -- 172.21.15.3:0/1236841183 reap_dead start 2024-05-21T17:21:26.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 1 -- 172.21.15.3:0/1236841183 shutdown_connections 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 1 -- 172.21.15.3:0/1236841183 >> 172.21.15.3:0/1236841183 conn(0x7f34b40a7ea0 msgr2=0x7f34b40a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 1 -- 172.21.15.3:0/1236841183 shutdown_connections 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.516+0000 7f34bc381640 1 -- 172.21.15.3:0/1236841183 wait complete. 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.520+0000 7f34bc381640 10 monclient: build_initial_monmap 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.520+0000 7f34bc381640 10 monclient: monmap: 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:26.529 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:26.530 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:26.530 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:26.530 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:26.530 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:26.530 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:26.530 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:26.530 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:26.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.520+0000 7f34bc381640 1 Processor -- start 2024-05-21T17:21:26.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.520+0000 7f34bc381640 1 -- start start 2024-05-21T17:21:26.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.520+0000 7f34bc381640 10 monclient: init 2024-05-21T17:21:26.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:26.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient: _add_conns ranks=[2,6,0,8,1,3,4,5,7] 2024-05-21T17:21:26.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient(hunting): picked mon.noname-b con 0x7f34b4136ac0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient(hunting): picked mon.noname-d con 0x7f34b41cdf00 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient(hunting): picked mon.noname-a con 0x7f34b41d46a0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b41264a0 con 0x7f34b41d46a0 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b409e730 con 0x7f34b4136ac0 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b41241b0 con 0x7f34b41cdf00 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient(hunting): authenticate will time out at 953.684646s 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34b991c640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f34b41cdf00 0x7f34b41d2f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:37038/0 (socket says 172.21.15.3:37038) 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34b991c640 1 -- 172.21.15.3:0/2257368502 learned_addr learned my addr 172.21.15.3:0/2257368502 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895749473 0 0) 0x7f34b41241b0 con 0x7f34b41cdf00 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:26.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): my global_id is 4110 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3498006630 con 0x7f34b41cdf00 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340495838 0 0) 0x7f34b41264a0 con 0x7f34b41d46a0 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): my global_id is 4104 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34b41241b0 con 0x7f34b41d46a0 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3241080127 0 0) 0x7f34b41241b0 con 0x7f34b41d46a0 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34b41264a0 con 0x7f34b41d46a0 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f34ac001c80 con 0x7f34b41d46a0 2024-05-21T17:21:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): dump: 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3793399930 0 0) 0x7f34b41264a0 con 0x7f34b41d46a0 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 >> v1:172.21.15.3:6792/0 conn(0x7f34b41cdf00 legacy=0x7f34b41d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 >> v1:172.21.15.3:6790/0 conn(0x7f34b4136ac0 legacy=0x7f34b4136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 monclient: found mon.a 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b41261c0 con 0x7f34b41d46a0 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 5 monclient: authenticate success, global_id 4104 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient: _renew_subs 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34b41ce320 con 0x7f34b41d46a0 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient: _renew_subs 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f34b41375b0 con 0x7f34b41d46a0 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f34ac0039a0 con 0x7f34b41d46a0 2024-05-21T17:21:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f34ac0020d0 con 0x7f34b41d46a0 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34b891a640 4 set_mon_vals no callback set 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.524+0000 7f34aaffd640 10 monclient: dump: 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.528+0000 7f34bc381640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.528+0000 7f34bc381640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.528+0000 7f34bc381640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.528+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f349c005090 con 0x7f34b41d46a0 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.528+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7f34ac004290 con 0x7f34b41d46a0 2024-05-21T17:21:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.528+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2082866315 0 0) 0x7f34ac0069c0 con 0x7f34b41d46a0 2024-05-21T17:21:26.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.532+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7f34ac004580 con 0x7f34b41d46a0 2024-05-21T17:21:26.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.532+0000 7f34aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:26.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.532+0000 7f34aaffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:26.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.620+0000 7f34bc381640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-21T17:21:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.620+0000 7f34bc381640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-21T17:21:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.620+0000 7f34bc381640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.620+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f349c005380 con 0x7f34b41d46a0 2024-05-21T17:21:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.624+0000 7f34aaffd640 1 -- 172.21.15.3:0/2257368502 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7f34ac004760 con 0x7f34b41d46a0 2024-05-21T17:21:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.624+0000 7f34aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-21T17:21:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.624+0000 7f34aaffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-05-21T17:21:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-05-21T17:21:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.628+0000 7f34bc381640 10 monclient: shutdown 2024-05-21T17:21:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.628+0000 7f34bc381640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.628+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 >> v1:172.21.15.3:6789/0 conn(0x7f34b41d46a0 legacy=0x7f34b41d6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.628+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 shutdown_connections 2024-05-21T17:21:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.628+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 >> 172.21.15.3:0/2257368502 conn(0x7f34b40a7ea0 msgr2=0x7f34b4139210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.628+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 shutdown_connections 2024-05-21T17:21:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:26.628+0000 7f34bc381640 1 -- 172.21.15.3:0/2257368502 wait complete. 2024-05-21T17:21:26.649 INFO:tasks.ceph:check_enable_crimson: False 2024-05-21T17:21:26.650 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-21T17:21:26.650 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:21:26.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-21T17:21:26.662 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:21:26.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-21T17:21:26.714 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-21T17:21:26.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-21T17:21:26.767 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:21:26.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-21T17:21:26.781 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:21:26.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-21T17:21:26.826 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:26.826+0000 7f8c3e7eb180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T17:21:26.838 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-05-21T17:21:26.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-21T17:21:26.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 7dfaea0b-2159-4f6e-86a6-da6637052771 0 2024-05-21T17:21:26.992 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:26.994+0000 7f8c3e7eb180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T17:21:27.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient: get_monmap_and_config 2024-05-21T17:21:27.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient: monmap: 2024-05-21T17:21:27.053 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.053 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.053 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:21:27.056932+0000 2024-05-21T17:21:27.053 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:21:27.056932+0000 2024-05-21T17:21:27.053 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.054 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.055 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.055 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 1 Processor -- start 2024-05-21T17:21:27.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 1 -- start start 2024-05-21T17:21:27.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient: init 2024-05-21T17:21:27.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient: _add_conns ranks=[1,0,3,4,5,2,7,8,6] 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient(hunting): picked mon.noname-f con 0x7f89ec0682d0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient(hunting): picked mon.noname-a con 0x7f89ec1373a0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient(hunting): picked mon.noname-g con 0x7f89ec136750 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89ec09bd20 con 0x7f89ec1373a0 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89ec0a0b20 con 0x7f89ec0682d0 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89ec09b410 con 0x7f89ec136750 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89f1838640 10 monclient(hunting): authenticate will time out at 942.123624s 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89eaffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f89ec0682d0 0x7f89ec136c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:35752/0 (socket says 172.21.15.203:35752) 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.054+0000 7f89eaffd640 1 -- 172.21.15.203:0/1488930244 learned_addr learned my addr 172.21.15.203:0/1488930244 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1535944181 0 0) 0x7f89ec0a0b20 con 0x7f89ec0682d0 2024-05-21T17:21:27.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient(hunting): my global_id is 4105 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d8003530 con 0x7f89ec0682d0 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888870240 0 0) 0x7f89ec09b410 con 0x7f89ec136750 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient(hunting): my global_id is 4116 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89ec0a0b20 con 0x7f89ec136750 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2986039381 0 0) 0x7f89ec0a0b20 con 0x7f89ec136750 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 >> v1:172.21.15.203:6789/0 conn(0x7f89ec0682d0 legacy=0x7f89ec136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 >> v1:172.21.15.3:6789/0 conn(0x7f89ec1373a0 legacy=0x7f89ec13ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 monclient: found mon.noname-g 2024-05-21T17:21:27.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.203:6790/0 2024-05-21T17:21:27.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 --> v1:172.21.15.203:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89ec0a3ad0 con 0x7f89ec136750 2024-05-21T17:21:27.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 5 monclient: authenticate success, global_id 4116 2024-05-21T17:21:27.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:27.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f89e4001ba0 con 0x7f89ec136750 2024-05-21T17:21:27.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: dump: 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 <== mon.3 v1:172.21.15.203:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89e4001dc0 con 0x7f89ec136750 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 1 -- 172.21.15.203:0/1488930244 <== mon.3 v1:172.21.15.203:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f89e4003ce0 con 0x7f89ec136750 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e9ffb640 10 monclient: dump: 2024-05-21T17:21:27.067 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.068 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.069 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:27.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 4 set_mon_vals no callback set 2024-05-21T17:21:27.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: shutdown 2024-05-21T17:21:27.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- 172.21.15.203:0/1488930244 >> v1:172.21.15.203:6790/0 conn(0x7f89ec136750 legacy=0x7f89ec13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- 172.21.15.203:0/1488930244 shutdown_connections 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- 172.21.15.203:0/1488930244 >> 172.21.15.203:0/1488930244 conn(0x7f89ec0a7640 msgr2=0x7f89ec0a7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- 172.21.15.203:0/1488930244 shutdown_connections 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- 172.21.15.203:0/1488930244 wait complete. 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: monmap: 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.070 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 Processor -- start 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- start start 2024-05-21T17:21:27.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: init 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: _add_conns ranks=[7,2,5,0,3,6,8,4,1] 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient(hunting): picked mon.noname-i con 0x7f89ec0682d0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient(hunting): picked mon.noname-b con 0x7f89ec136750 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient(hunting): picked mon.noname-h con 0x7f89ec1373a0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89ec072f50 con 0x7f89ec136750 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89ec1cc960 con 0x7f89ec1373a0 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89ec09de80 con 0x7f89ec0682d0 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient(hunting): authenticate will time out at 942.126915s 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89eaffd640 1 --1- >> v1:172.21.15.203:6792/0 conn(0x7f89ec0682d0 0x7f89ec1cbb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6792/0 says I am v1:172.21.15.203:41326/0 (socket says 172.21.15.203:41326) 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89eaffd640 1 -- 172.21.15.203:0/450794416 learned_addr learned my addr 172.21.15.203:0/450794416 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3432421704 0 0) 0x7f89ec09de80 con 0x7f89ec0682d0 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): my global_id is 4120 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89cc003530 con 0x7f89ec0682d0 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4167829320 0 0) 0x7f89ec1cc960 con 0x7f89ec1373a0 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): my global_id is 4136 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89ec09de80 con 0x7f89ec1373a0 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1303290074 0 0) 0x7f89cc003530 con 0x7f89ec0682d0 2024-05-21T17:21:27.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89ec1cc960 con 0x7f89ec0682d0 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f89d40019e0 con 0x7f89ec0682d0 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): dump: 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.074 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 701013438 0 0) 0x7f89ec09de80 con 0x7f89ec1373a0 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89cc003530 con 0x7f89ec1373a0 2024-05-21T17:21:27.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f89e4001b60 con 0x7f89ec1373a0 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): dump: 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.076 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777423224 0 0) 0x7f89ec072f50 con 0x7f89ec136750 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient(hunting): my global_id is 4106 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89ec09de80 con 0x7f89ec136750 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1076243815 0 0) 0x7f89ec1cc960 con 0x7f89ec0682d0 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 >> v1:172.21.15.203:6791/0 conn(0x7f89ec1373a0 legacy=0x7f89ec1d3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 >> v1:172.21.15.3:6790/0 conn(0x7f89ec136750 legacy=0x7f89ec1cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 monclient: found mon.i 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.203:6792/0 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89ec09dac0 con 0x7f89ec0682d0 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 5 monclient: authenticate success, global_id 4120 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: _renew_subs 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.203:6792/0 2024-05-21T17:21:27.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.058+0000 7f89f1838640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89ec073360 con 0x7f89ec0682d0 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89f1838640 10 monclient: _renew_subs 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89f1838640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.203:6792/0 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89f1838640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f89ec072040 con 0x7f89ec0682d0 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89d4001ca0 con 0x7f89ec0682d0 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f89d4004de0 con 0x7f89ec0682d0 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e97fa640 4 set_mon_vals no callback set 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 10 monclient: dump: 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.079 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7f89d4005380 con 0x7f89ec0682d0 2024-05-21T17:21:27.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 495076125 0 0) 0x7f89d4006870 con 0x7f89ec0682d0 2024-05-21T17:21:27.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e1ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e1ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e1ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.203:6792/0 2024-05-21T17:21:27.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e1ffb640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c4005090 con 0x7f89ec0682d0 2024-05-21T17:21:27.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7f89d4006c20 con 0x7f89ec0682d0 2024-05-21T17:21:27.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.062+0000 7f89e3fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:27.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.130+0000 7f89e1ffb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7dfaea0b-2159-4f6e-86a6-da6637052771", "id": 0}] 2024-05-21T17:21:27.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.130+0000 7f89e1ffb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7dfaea0b-2159-4f6e-86a6-da6637052771", "id": 0}] 2024-05-21T17:21:27.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.130+0000 7f89e1ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.203:6792/0 2024-05-21T17:21:27.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.130+0000 7f89e1ffb640 1 -- 172.21.15.203:0/450794416 --> v1:172.21.15.203:6792/0 -- mon_command({"prefix": "osd new", "uuid": "7dfaea0b-2159-4f6e-86a6-da6637052771", "id": 0} v 0) -- 0x7f89c4005380 con 0x7f89ec0682d0 2024-05-21T17:21:27.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e3fff640 1 -- 172.21.15.203:0/450794416 <== mon.7 v1:172.21.15.203:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7dfaea0b-2159-4f6e-86a6-da6637052771", "id": 0}]=0 v3) ==== 112+0+2 (unknown 2665055035 0 732707515) 0x7f89d4005850 con 0x7f89ec0682d0 2024-05-21T17:21:27.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7dfaea0b-2159-4f6e-86a6-da6637052771", "id": 0}] 2024-05-21T17:21:27.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e3fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:27.140 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-05-21T17:21:27.141 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e1ffb640 10 monclient: shutdown 2024-05-21T17:21:27.141 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e1ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.141 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e1ffb640 1 -- 172.21.15.203:0/450794416 >> v1:172.21.15.203:6792/0 conn(0x7f89ec0682d0 legacy=0x7f89ec1cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e1ffb640 1 -- 172.21.15.203:0/450794416 shutdown_connections 2024-05-21T17:21:27.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e1ffb640 1 -- 172.21.15.203:0/450794416 >> 172.21.15.203:0/450794416 conn(0x7f89ec0a7640 msgr2=0x7f89ec094450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e1ffb640 1 -- 172.21.15.203:0/450794416 shutdown_connections 2024-05-21T17:21:27.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.142+0000 7f89e1ffb640 1 -- 172.21.15.203:0/450794416 wait complete. 2024-05-21T17:21:27.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 28d053ac-034d-43ed-a83e-9ee62a426b88 1 2024-05-21T17:21:27.214 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:27.214+0000 7f8c3e7eb180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T17:21:27.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.278+0000 7fd70072a640 10 monclient: get_monmap_and_config 2024-05-21T17:21:27.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.278+0000 7fd70072a640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.278+0000 7fd70072a640 10 monclient: monmap: 2024-05-21T17:21:27.278 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.278 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:21:27.282002+0000 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:21:27.282002+0000 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.279 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.280 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.280 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.280 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.278+0000 7fd70072a640 1 Processor -- start 2024-05-21T17:21:27.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.278+0000 7fd70072a640 1 -- start start 2024-05-21T17:21:27.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.278+0000 7fd70072a640 10 monclient: init 2024-05-21T17:21:27.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: _add_conns ranks=[1,2,4,8,5,7,0,6,3] 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): picked mon.noname-f con 0x7fd6f80682d0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f8136750 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): picked mon.noname-c con 0x7fd6f81372f0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f80a14d0 con 0x7fd6f80682d0 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f80686b0 con 0x7fd6f8136750 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f809dc40 con 0x7fd6f81372f0 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): authenticate will time out at 942.348587s 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fe4c6640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd6f80682d0 0x7fd6f8136be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:35760/0 (socket says 172.21.15.203:35760) 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fe4c6640 1 -- 172.21.15.203:0/913272658 learned_addr learned my addr 172.21.15.203:0/913272658 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1494245450 0 0) 0x7fd6f80a14d0 con 0x7fd6f80682d0 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient(hunting): my global_id is 4114 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6e4003530 con 0x7fd6f80682d0 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3129817154 0 0) 0x7fd6e4003530 con 0x7fd6f80682d0 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 >> v1:172.21.15.3:6791/0 conn(0x7fd6f81372f0 legacy=0x7fd6f813f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 >> v1:172.21.15.3:6790/0 conn(0x7fd6f8136750 legacy=0x7fd6f813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 monclient: found mon.noname-f 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.203:6789/0 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f809b660 con 0x7fd6f80682d0 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 5 monclient: authenticate success, global_id 4114 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd6f0001bc0 con 0x7fd6f80682d0 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: dump: 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.289 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6f0001de0 con 0x7fd6f80682d0 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 1 -- 172.21.15.203:0/913272658 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd6f0004f70 con 0x7fd6f80682d0 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 4 set_mon_vals no callback set 2024-05-21T17:21:27.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fd4c4640 10 monclient: dump: 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.291 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: shutdown 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- 172.21.15.203:0/913272658 >> v1:172.21.15.203:6789/0 conn(0x7fd6f80682d0 legacy=0x7fd6f8136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- 172.21.15.203:0/913272658 shutdown_connections 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- 172.21.15.203:0/913272658 >> 172.21.15.203:0/913272658 conn(0x7fd6f80a7640 msgr2=0x7fd6f80a7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- 172.21.15.203:0/913272658 shutdown_connections 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- 172.21.15.203:0/913272658 wait complete. 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: monmap: 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.293 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 Processor -- start 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- start start 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: init 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient: _add_conns ranks=[5,2,6,1,0,8,7,3,4] 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): picked mon.noname-h con 0x7fd6f80682d0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f81372f0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): picked mon.noname-d con 0x7fd6f8096350 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f809da30 con 0x7fd6f81372f0 2024-05-21T17:21:27.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f8095a70 con 0x7fd6f80682d0 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f809de80 con 0x7fd6f8096350 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd70072a640 10 monclient(hunting): authenticate will time out at 942.351577s 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fe4c6640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7fd6f80682d0 0x7fd6f8135820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.203:55630/0 (socket says 172.21.15.203:55630) 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6fe4c6640 1 -- 172.21.15.203:0/3605746283 learned_addr learned my addr 172.21.15.203:0/3605746283 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2551447590 0 0) 0x7fd6f8095a70 con 0x7fd6f80682d0 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6eaffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6eaffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6eaffd640 10 monclient(hunting): my global_id is 4145 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.282+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6dc003530 con 0x7fd6f80682d0 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2842750804 0 0) 0x7fd6dc003530 con 0x7fd6f80682d0 2024-05-21T17:21:27.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6f8095a70 con 0x7fd6f80682d0 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd6f0001f90 con 0x7fd6f80682d0 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): dump: 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.296 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221913106 0 0) 0x7fd6f80686b0 con 0x7fd6f81372f0 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): my global_id is 4124 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6dc003530 con 0x7fd6f81372f0 2024-05-21T17:21:27.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455783820 0 0) 0x7fd6f809de80 con 0x7fd6f8096350 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient(hunting): my global_id is 4119 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f80686b0 con 0x7fd6f8096350 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1781554248 0 0) 0x7fd6f8095a70 con 0x7fd6f80682d0 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 >> v1:172.21.15.3:6792/0 conn(0x7fd6f8096350 legacy=0x7fd6f81d48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 >> v1:172.21.15.3:6790/0 conn(0x7fd6f81372f0 legacy=0x7fd6f81d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 monclient: found mon.h 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:21:27.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.203:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f80747c0 con 0x7fd6f80682d0 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 5 monclient: authenticate success, global_id 4145 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 10 monclient: _renew_subs 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.203:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f8073170 con 0x7fd6f80682d0 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 10 monclient: _renew_subs 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.203:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd6f80720c0 con 0x7fd6f80682d0 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6f0005160 con 0x7fd6f80682d0 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6fccc3640 4 set_mon_vals no callback set 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd6f0005650 con 0x7fd6f80682d0 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: dump: 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.300 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7fd6f0005b70 con 0x7fd6f80682d0 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 2660779775 0 0) 0x7fd6f0006d50 con 0x7fd6f80682d0 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:21:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.203:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e0005090 con 0x7fd6f80682d0 2024-05-21T17:21:27.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7fd6f0001d00 con 0x7fd6f80682d0 2024-05-21T17:21:27.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.286+0000 7fd6eaffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:27.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.358+0000 7fd70072a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "28d053ac-034d-43ed-a83e-9ee62a426b88", "id": 1}] 2024-05-21T17:21:27.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.358+0000 7fd70072a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "28d053ac-034d-43ed-a83e-9ee62a426b88", "id": 1}] 2024-05-21T17:21:27.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.358+0000 7fd70072a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:21:27.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.358+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 --> v1:172.21.15.203:6791/0 -- mon_command({"prefix": "osd new", "uuid": "28d053ac-034d-43ed-a83e-9ee62a426b88", "id": 1} v 0) -- 0x7fd6e0005380 con 0x7fd6f80682d0 2024-05-21T17:21:27.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd6eaffd640 1 -- 172.21.15.203:0/3605746283 <== mon.5 v1:172.21.15.203:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "28d053ac-034d-43ed-a83e-9ee62a426b88", "id": 1}]=0 v4) ==== 112+0+2 (unknown 1052167163 0 940483276) 0x7fd6f0005f30 con 0x7fd6f80682d0 2024-05-21T17:21:27.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd6eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "28d053ac-034d-43ed-a83e-9ee62a426b88", "id": 1}] 2024-05-21T17:21:27.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd6eaffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-05-21T17:21:27.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd70072a640 10 monclient: shutdown 2024-05-21T17:21:27.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd70072a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 >> v1:172.21.15.203:6791/0 conn(0x7fd6f80682d0 legacy=0x7fd6f8135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 shutdown_connections 2024-05-21T17:21:27.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 >> 172.21.15.203:0/3605746283 conn(0x7fd6f80a7640 msgr2=0x7fd6f80944e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 shutdown_connections 2024-05-21T17:21:27.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.366+0000 7fd70072a640 1 -- 172.21.15.203:0/3605746283 wait complete. 2024-05-21T17:21:27.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 9de90871-f66b-4a85-ac48-772541c5b23d 2 2024-05-21T17:21:27.495 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: get_monmap_and_config 2024-05-21T17:21:27.495 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: monmap: 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:21:27.499284+0000 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:21:27.499284+0000 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.496 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.497 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.497 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.497 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.497 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.497 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.497 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.497 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 Processor -- start 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- start start 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: init 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: _add_conns ranks=[8,5,7,1,0,6,4,3,2] 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient(hunting): picked mon.noname-e con 0x7f03c8135b60 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient(hunting): picked mon.noname-h con 0x7f03c809dba0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient(hunting): picked mon.noname-i con 0x7f03c813f280 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03c806c580 con 0x7f03c809dba0 2024-05-21T17:21:27.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03c809cee0 con 0x7f03c813f280 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03c809d1c0 con 0x7f03c8135b60 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient(hunting): authenticate will time out at 942.565700s 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c5d9b640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f03c809dba0 0x7f03c813db60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.203:55646/0 (socket says 172.21.15.203:55646) 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c6d9d640 1 --1- >> v1:172.21.15.203:6792/0 conn(0x7f03c813f280 0x7f03c8141670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6792/0 says I am v1:172.21.15.203:41336/0 (socket says 172.21.15.203:41336) 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c5d9b640 1 -- 172.21.15.203:0/442518009 learned_addr learned my addr 172.21.15.203:0/442518009 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1090416269 0 0) 0x7f03c809cee0 con 0x7f03c813f280 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient(hunting): my global_id is 4129 2024-05-21T17:21:27.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03b4003530 con 0x7f03c813f280 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 733155525 0 0) 0x7f03c806c580 con 0x7f03c809dba0 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient(hunting): my global_id is 4154 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03c809cee0 con 0x7f03c809dba0 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4276638189 0 0) 0x7f03c809cee0 con 0x7f03c809dba0 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 >> v1:172.21.15.3:6793/0 conn(0x7f03c8135b60 legacy=0x7f03c8135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 >> v1:172.21.15.203:6792/0 conn(0x7f03c813f280 legacy=0x7f03c8141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 monclient: found mon.noname-h 2024-05-21T17:21:27.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.203:6791/0 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 --> v1:172.21.15.203:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c806d120 con 0x7f03c809dba0 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f03b0001820 con 0x7f03c809dba0 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: dump: 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.508 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 5 monclient: authenticate success, global_id 4154 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 <== mon.5 v1:172.21.15.203:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f03b0001a40 con 0x7f03c809dba0 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 1 -- 172.21.15.203:0/442518009 <== mon.5 v1:172.21.15.203:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f03b0004bf0 con 0x7f03c809dba0 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 4 set_mon_vals no callback set 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03c559a640 10 monclient: dump: 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.510 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: shutdown 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- 172.21.15.203:0/442518009 >> v1:172.21.15.203:6791/0 conn(0x7f03c809dba0 legacy=0x7f03c813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- 172.21.15.203:0/442518009 shutdown_connections 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- 172.21.15.203:0/442518009 >> 172.21.15.203:0/442518009 conn(0x7f03c80a84e0 msgr2=0x7f03c80a49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- 172.21.15.203:0/442518009 shutdown_connections 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 1 -- 172.21.15.203:0/442518009 wait complete. 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.498+0000 7f03ccbb6640 10 monclient: monmap: 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.512 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 1 Processor -- start 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 1 -- start start 2024-05-21T17:21:27.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: init 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: _add_conns ranks=[4,5,3,2,0,1,7,8,6] 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient(hunting): picked mon.noname-c con 0x7f03c809dba0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient(hunting): picked mon.noname-h con 0x7f03c8135b60 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient(hunting): picked mon.noname-g con 0x7f03c813f280 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03c81cabd0 con 0x7f03c813f280 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03c809e8a0 con 0x7f03c809dba0 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03c806d3b0 con 0x7f03c8135b60 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient(hunting): authenticate will time out at 942.568776s 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03c5d9b640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f03c8135b60 0x7f03c81ca4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.203:55658/0 (socket says 172.21.15.203:55658) 2024-05-21T17:21:27.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03c5d9b640 1 -- 172.21.15.203:0/1364063519 learned_addr learned my addr 172.21.15.203:0/1364063519 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2678668619 0 0) 0x7f03c806d3b0 con 0x7f03c8135b60 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): my global_id is 4163 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03a8003530 con 0x7f03c8135b60 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3743177071 0 0) 0x7f03c81cabd0 con 0x7f03c813f280 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): my global_id is 4125 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03c806d3b0 con 0x7f03c813f280 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 765130981 0 0) 0x7f03c806d3b0 con 0x7f03c813f280 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03c81cabd0 con 0x7f03c813f280 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f03c0001ca0 con 0x7f03c813f280 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): dump: 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.516 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:27.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 823588487 0 0) 0x7f03a8003530 con 0x7f03c8135b60 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03c806d3b0 con 0x7f03c8135b60 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f03b0004b30 con 0x7f03c8135b60 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): dump: 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 385723214 0 0) 0x7f03c809e8a0 con 0x7f03c809dba0 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient(hunting): my global_id is 4135 2024-05-21T17:21:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03a8003530 con 0x7f03c809dba0 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 829303749 0 0) 0x7f03c81cabd0 con 0x7f03c813f280 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 >> v1:172.21.15.203:6791/0 conn(0x7f03c8135b60 legacy=0x7f03c81ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 >> v1:172.21.15.3:6791/0 conn(0x7f03c809dba0 legacy=0x7f03c81c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 monclient: found mon.g 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c8131900 con 0x7f03c813f280 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 5 monclient: authenticate success, global_id 4125 2024-05-21T17:21:27.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: _renew_subs 2024-05-21T17:21:27.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:27.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f03c812fd60 con 0x7f03c813f280 2024-05-21T17:21:27.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: _renew_subs 2024-05-21T17:21:27.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:27.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f03c81302d0 con 0x7f03c813f280 2024-05-21T17:21:27.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f03c0004b70 con 0x7f03c813f280 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f03c0005060 con 0x7f03c813f280 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 10 monclient: dump: 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.522 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.523 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03c4d99640 4 set_mon_vals no callback set 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7f03c0005580 con 0x7f03c813f280 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 1331858682 0 0) 0x7f03c0006ba0 con 0x7f03c813f280 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.502+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c80698d0 con 0x7f03c813f280 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.506+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7f03c0006f50 con 0x7f03c813f280 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.506+0000 7f03beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.506+0000 7f03beffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:27.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.578+0000 7f03ccbb6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9de90871-f66b-4a85-ac48-772541c5b23d", "id": 2}] 2024-05-21T17:21:27.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.578+0000 7f03ccbb6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9de90871-f66b-4a85-ac48-772541c5b23d", "id": 2}] 2024-05-21T17:21:27.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.578+0000 7f03ccbb6640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:27.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.578+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 --> v1:172.21.15.203:6790/0 -- mon_command({"prefix": "osd new", "uuid": "9de90871-f66b-4a85-ac48-772541c5b23d", "id": 2} v 0) -- 0x7f03c8142490 con 0x7f03c813f280 2024-05-21T17:21:27.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03beffd640 1 -- 172.21.15.203:0/1364063519 <== mon.3 v1:172.21.15.203:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9de90871-f66b-4a85-ac48-772541c5b23d", "id": 2}]=0 v5) ==== 112+0+2 (unknown 1345997914 0 216599125) 0x7f03c0006850 con 0x7f03c813f280 2024-05-21T17:21:27.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9de90871-f66b-4a85-ac48-772541c5b23d", "id": 2}] 2024-05-21T17:21:27.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03beffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:27.582 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-05-21T17:21:27.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03ccbb6640 10 monclient: shutdown 2024-05-21T17:21:27.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03ccbb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 >> v1:172.21.15.203:6790/0 conn(0x7f03c813f280 legacy=0x7f03c81d5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 shutdown_connections 2024-05-21T17:21:27.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 >> 172.21.15.203:0/1364063519 conn(0x7f03c80a84e0 msgr2=0x7f03c813f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 shutdown_connections 2024-05-21T17:21:27.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.582+0000 7f03ccbb6640 1 -- 172.21.15.203:0/1364063519 wait complete. 2024-05-21T17:21:27.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new f8431d10-6318-458b-9a94-51fc47273f87 3 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient: get_monmap_and_config 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient: monmap: 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:21:27.711742+0000 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:21:27.711742+0000 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.708 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.709 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 1 Processor -- start 2024-05-21T17:21:27.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 1 -- start start 2024-05-21T17:21:27.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient: init 2024-05-21T17:21:27.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient: _add_conns ranks=[2,0,4,5,1,6,7,8,3] 2024-05-21T17:21:27.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient(hunting): picked mon.noname-b con 0x7f8b240a38f0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient(hunting): picked mon.noname-a con 0x7f8b240a3cd0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient(hunting): picked mon.noname-c con 0x7f8b2413bb00 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2409e510 con 0x7f8b240a3cd0 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2409d1b0 con 0x7f8b240a38f0 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b240a0140 con 0x7f8b2413bb00 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient(hunting): authenticate will time out at 942.778121s 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2989b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8b240a3cd0 0x7f8b240a40e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.203:52996/0 (socket says 172.21.15.203:52996) 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2a89d640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f8b2413bb00 0x7f8b2413def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.203:39634/0 (socket says 172.21.15.203:39634) 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2989b640 1 -- 172.21.15.203:0/3872513241 learned_addr learned my addr 172.21.15.203:0/3872513241 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395909565 0 0) 0x7f8b2409d1b0 con 0x7f8b240a38f0 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): my global_id is 4133 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b10003530 con 0x7f8b240a38f0 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2804795481 0 0) 0x7f8b2409e510 con 0x7f8b240a3cd0 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): my global_id is 4122 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b2409d1b0 con 0x7f8b240a3cd0 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3743618515 0 0) 0x7f8b240a0140 con 0x7f8b2413bb00 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient(hunting): my global_id is 4144 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b2409e510 con 0x7f8b2413bb00 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 406654238 0 0) 0x7f8b10003530 con 0x7f8b240a38f0 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 >> v1:172.21.15.3:6791/0 conn(0x7f8b2413bb00 legacy=0x7f8b2413def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 >> v1:172.21.15.3:6789/0 conn(0x7f8b240a3cd0 legacy=0x7f8b240a40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 monclient: found mon.noname-b 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b240a5350 con 0x7f8b240a38f0 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f8b20001770 con 0x7f8b240a38f0 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: dump: 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.717 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 5 monclient: authenticate success, global_id 4133 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b20001990 con 0x7f8b240a38f0 2024-05-21T17:21:27.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2c300640 4 set_mon_vals no callback set 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 1 -- 172.21.15.203:0/3872513241 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f8b20003960 con 0x7f8b240a38f0 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.710+0000 7f8b2909a640 10 monclient: dump: 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.719 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: shutdown 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- 172.21.15.203:0/3872513241 >> v1:172.21.15.3:6790/0 conn(0x7f8b240a38f0 legacy=0x7f8b2409fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- 172.21.15.203:0/3872513241 shutdown_connections 2024-05-21T17:21:27.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- 172.21.15.203:0/3872513241 >> 172.21.15.203:0/3872513241 conn(0x7f8b240a84e0 msgr2=0x7f8b240a49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- 172.21.15.203:0/3872513241 shutdown_connections 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- 172.21.15.203:0/3872513241 wait complete. 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: monmap: 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.721 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 Processor -- start 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- start start 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: init 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: _add_conns ranks=[0,4,8,6,1,2,5,7,3] 2024-05-21T17:21:27.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient(hunting): picked mon.noname-a con 0x7f8b240a38f0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient(hunting): picked mon.noname-c con 0x7f8b240a3cd0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient(hunting): picked mon.noname-e con 0x7f8b2413bb00 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b24072e30 con 0x7f8b240a38f0 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2413df20 con 0x7f8b240a3cd0 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b24074100 con 0x7f8b2413bb00 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient(hunting): authenticate will time out at 942.782028s 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2989b640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f8b240a3cd0 0x7f8b2413d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.203:39640/0 (socket says 172.21.15.203:39640) 2024-05-21T17:21:27.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2989b640 1 -- 172.21.15.203:0/43341289 learned_addr learned my addr 172.21.15.203:0/43341289 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4163639049 0 0) 0x7f8b2413df20 con 0x7f8b240a3cd0 2024-05-21T17:21:27.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): my global_id is 4153 2024-05-21T17:21:27.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b08003530 con 0x7f8b240a3cd0 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1723029837 0 0) 0x7f8b24074100 con 0x7f8b2413bb00 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): my global_id is 4130 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b2413df20 con 0x7f8b2413bb00 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460184437 0 0) 0x7f8b24072e30 con 0x7f8b240a38f0 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): my global_id is 4131 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b24074100 con 0x7f8b240a38f0 2024-05-21T17:21:27.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1835356062 0 0) 0x7f8b2413df20 con 0x7f8b2413bb00 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b24072e30 con 0x7f8b2413bb00 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f8b1c001d00 con 0x7f8b2413bb00 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): dump: 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.728 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 886558937 0 0) 0x7f8b24074100 con 0x7f8b240a38f0 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b2413df20 con 0x7f8b240a38f0 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1674512212 0 0) 0x7f8b08003530 con 0x7f8b240a3cd0 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b24074100 con 0x7f8b240a3cd0 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f8b14001ce0 con 0x7f8b240a3cd0 2024-05-21T17:21:27.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): dump: 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.730 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f8b20003940 con 0x7f8b240a38f0 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient(hunting): dump: 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.731 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1687276961 0 0) 0x7f8b24072e30 con 0x7f8b2413bb00 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 >> v1:172.21.15.3:6791/0 conn(0x7f8b240a3cd0 legacy=0x7f8b2413d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 >> v1:172.21.15.3:6789/0 conn(0x7f8b240a38f0 legacy=0x7f8b2413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 monclient: found mon.e 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2409d720 con 0x7f8b2413bb00 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 5 monclient: authenticate success, global_id 4130 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: _renew_subs 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b240722d0 con 0x7f8b2413bb00 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: _renew_subs 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:27.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.714+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b24072b50 con 0x7f8b2413bb00 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b1c004b70 con 0x7f8b2413bb00 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b1affd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f8b1c0050e0 con 0x7f8b2413bb00 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b28899640 4 set_mon_vals no callback set 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b1affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b1affd640 10 monclient: dump: 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.734 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7f8b1c005680 con 0x7f8b2413bb00 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 3066946063 0 0) 0x7f8b1c006de0 con 0x7f8b2413bb00 2024-05-21T17:21:27.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b2c300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b2c300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b2c300640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:27.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.718+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b0c005090 con 0x7f8b2413bb00 2024-05-21T17:21:27.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.722+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7f8b1c006a00 con 0x7f8b2413bb00 2024-05-21T17:21:27.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.722+0000 7f8b1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.722+0000 7f8b1affd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:27.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.790+0000 7f8b2c300640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f8431d10-6318-458b-9a94-51fc47273f87", "id": 3}] 2024-05-21T17:21:27.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.790+0000 7f8b2c300640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f8431d10-6318-458b-9a94-51fc47273f87", "id": 3}] 2024-05-21T17:21:27.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.790+0000 7f8b2c300640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:27.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.790+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "osd new", "uuid": "f8431d10-6318-458b-9a94-51fc47273f87", "id": 3} v 0) -- 0x7f8b0c005380 con 0x7f8b2413bb00 2024-05-21T17:21:27.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b1affd640 1 -- 172.21.15.203:0/43341289 <== mon.8 v1:172.21.15.3:6793/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f8431d10-6318-458b-9a94-51fc47273f87", "id": 3}]=0 v6) ==== 112+0+2 (unknown 1302970360 0 525046306) 0x7f8b1c005ba0 con 0x7f8b2413bb00 2024-05-21T17:21:27.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f8431d10-6318-458b-9a94-51fc47273f87", "id": 3}] 2024-05-21T17:21:27.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b1affd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:27.800 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-05-21T17:21:27.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b2c300640 10 monclient: shutdown 2024-05-21T17:21:27.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b2c300640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 >> v1:172.21.15.3:6793/0 conn(0x7f8b2413bb00 legacy=0x7f8b241d3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 shutdown_connections 2024-05-21T17:21:27.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 >> 172.21.15.203:0/43341289 conn(0x7f8b240a84e0 msgr2=0x7f8b2406bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 shutdown_connections 2024-05-21T17:21:27.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.802+0000 7f8b2c300640 1 -- 172.21.15.203:0/43341289 wait complete. 2024-05-21T17:21:27.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 028748a9-3ca4-4730-960e-8913c9504a9d 4 2024-05-21T17:21:27.826 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:27.826+0000 7f8c3e7eb180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T17:21:27.891 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:27.894+0000 7f8c3e7eb180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T17:21:27.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient: get_monmap_and_config 2024-05-21T17:21:27.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient: monmap: 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:21:27.933463+0000 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:21:27.933463+0000 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.930 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.931 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.931 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.931 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.931 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.931 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 1 Processor -- start 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 1 -- start start 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient: init 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient: _add_conns ranks=[6,5,1,0,2,4,8,3,7] 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient(hunting): picked mon.noname-d con 0x7f67300a6220 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient(hunting): picked mon.noname-h con 0x7f6730137380 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient(hunting): picked mon.noname-f con 0x7f67300a2fb0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67300a22f0 con 0x7f67300a2fb0 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67300a1600 con 0x7f6730137380 2024-05-21T17:21:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67300a0f80 con 0x7f67300a6220 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f6734cb4640 10 monclient(hunting): authenticate will time out at 942.999815s 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f672ed9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f67300a2fb0 0x7f673013f200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:35764/0 (socket says 172.21.15.203:35764) 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f672dd9b640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f6730137380 0x7f673013bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.203:55666/0 (socket says 172.21.15.203:55666) 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.930+0000 7f672ed9d640 1 -- 172.21.15.203:0/527667345 learned_addr learned my addr 172.21.15.203:0/527667345 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2679661800 0 0) 0x7f6720001120 con 0x7f67300a2fb0 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient(hunting): my global_id is 4123 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6718003530 con 0x7f67300a2fb0 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717374796 0 0) 0x7f67300a1600 con 0x7f6730137380 2024-05-21T17:21:27.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient(hunting): my global_id is 4172 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6720001120 con 0x7f6730137380 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1519169751 0 0) 0x7f6718003530 con 0x7f67300a2fb0 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 >> v1:172.21.15.3:6792/0 conn(0x7f67300a6220 legacy=0x7f6730136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 >> v1:172.21.15.203:6791/0 conn(0x7f6730137380 legacy=0x7f673013bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 monclient: found mon.noname-f 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.203:6789/0 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67300a25d0 con 0x7f67300a2fb0 2024-05-21T17:21:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6720001a10 con 0x7f67300a2fb0 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: dump: 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.940 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 5 monclient: authenticate success, global_id 4123 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6720001bb0 con 0x7f67300a2fb0 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 4 set_mon_vals no callback set 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 1 -- 172.21.15.203:0/527667345 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6720004bf0 con 0x7f67300a2fb0 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672d59a640 10 monclient: dump: 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.942 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: shutdown 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- 172.21.15.203:0/527667345 >> v1:172.21.15.203:6789/0 conn(0x7f67300a2fb0 legacy=0x7f673013f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- 172.21.15.203:0/527667345 shutdown_connections 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- 172.21.15.203:0/527667345 >> 172.21.15.203:0/527667345 conn(0x7f673006ccf0 msgr2=0x7f67300a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:27.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- 172.21.15.203:0/527667345 shutdown_connections 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- 172.21.15.203:0/527667345 wait complete. 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: build_initial_monmap 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: monmap: 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:27.944 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 Processor -- start 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- start start 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: init 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: _add_conns ranks=[0,1,5,7,3,6,4,2,8] 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient(hunting): picked mon.noname-a con 0x7f67300a2fb0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:27.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient(hunting): picked mon.noname-f con 0x7f67300a6220 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient(hunting): picked mon.noname-h con 0x7f6730137380 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67300a1820 con 0x7f67300a2fb0 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67300744c0 con 0x7f67300a6220 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f673006ec20 con 0x7f6730137380 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient(hunting): authenticate will time out at 943.002898s 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672ed9d640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f6730137380 0x7f67301d6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.203:55676/0 (socket says 172.21.15.203:55676) 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672dd9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f67300a6220 0x7f67301d2de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:35770/0 (socket says 172.21.15.203:35770) 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672ed9d640 1 -- 172.21.15.203:0/1019877431 learned_addr learned my addr 172.21.15.203:0/1019877431 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724452315 0 0) 0x7f67300744c0 con 0x7f67300a6220 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): my global_id is 4132 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6710003530 con 0x7f67300a6220 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444619956 0 0) 0x7f673006ec20 con 0x7f6730137380 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): my global_id is 4181 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67300744c0 con 0x7f6730137380 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4268936319 0 0) 0x7f6710003530 con 0x7f67300a6220 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f673006ec20 con 0x7f67300a6220 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6724001a00 con 0x7f67300a6220 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): dump: 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.948 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3163192417 0 0) 0x7f67300744c0 con 0x7f6730137380 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6710003530 con 0x7f6730137380 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6720004b30 con 0x7f6730137380 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient(hunting): dump: 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.949 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.950 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2026277995 0 0) 0x7f673006ec20 con 0x7f67300a6220 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 >> v1:172.21.15.203:6791/0 conn(0x7f6730137380 legacy=0x7f67301d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 >> v1:172.21.15.3:6789/0 conn(0x7f67300a2fb0 legacy=0x7f6730072d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 monclient: found mon.f 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.203:6789/0 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6730136500 con 0x7f67300a6220 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient: _finish_auth 0 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 5 monclient: authenticate success, global_id 4132 2024-05-21T17:21:27.951 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: _renew_subs 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.203:6789/0 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f673006e870 con 0x7f67300a6220 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: _renew_subs 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.203:6789/0 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f673006daa0 con 0x7f67300a6220 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6724001d00 con 0x7f67300a6220 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6724004de0 con 0x7f67300a6220 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f672cd99640 4 set_mon_vals no callback set 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 10 monclient: dump: 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:27.952 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:27.953 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:27.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7f6724005380 con 0x7f67300a6220 2024-05-21T17:21:27.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.934+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 1742574350 0 0) 0x7f6724006b10 con 0x7f67300a6220 2024-05-21T17:21:27.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.938+0000 7f6734cb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.938+0000 7f6734cb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.938+0000 7f6734cb4640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.203:6789/0 2024-05-21T17:21:27.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.938+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6714005090 con 0x7f67300a6220 2024-05-21T17:21:27.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.938+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7f67240066b0 con 0x7f67300a6220 2024-05-21T17:21:27.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.938+0000 7f671effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:27.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:27.938+0000 7f671effd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:27.955 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:27.946+0000 7f8c3e7eb180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T17:21:28.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.006+0000 7f6734cb4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "028748a9-3ca4-4730-960e-8913c9504a9d", "id": 4}] 2024-05-21T17:21:28.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.006+0000 7f6734cb4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "028748a9-3ca4-4730-960e-8913c9504a9d", "id": 4}] 2024-05-21T17:21:28.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.006+0000 7f6734cb4640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.203:6789/0 2024-05-21T17:21:28.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.006+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "028748a9-3ca4-4730-960e-8913c9504a9d", "id": 4} v 0) -- 0x7f6714005380 con 0x7f67300a6220 2024-05-21T17:21:28.013 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f671effd640 1 -- 172.21.15.203:0/1019877431 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "028748a9-3ca4-4730-960e-8913c9504a9d", "id": 4}]=0 v7) ==== 112+0+2 (unknown 2662530868 0 1697012583) 0x7f6724005850 con 0x7f67300a6220 2024-05-21T17:21:28.013 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f671effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "028748a9-3ca4-4730-960e-8913c9504a9d", "id": 4}] 2024-05-21T17:21:28.013 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f671effd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:28.013 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-05-21T17:21:28.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f6734cb4640 10 monclient: shutdown 2024-05-21T17:21:28.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f6734cb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:28.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 >> v1:172.21.15.203:6789/0 conn(0x7f67300a6220 legacy=0x7f67301d2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 shutdown_connections 2024-05-21T17:21:28.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 >> 172.21.15.203:0/1019877431 conn(0x7f673006ccf0 msgr2=0x7f673009ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:28.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 shutdown_connections 2024-05-21T17:21:28.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.014+0000 7f6734cb4640 1 -- 172.21.15.203:0/1019877431 wait complete. 2024-05-21T17:21:28.026 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:28.026+0000 7f8c3e7eb180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T17:21:28.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 87412b84-97f1-4350-9034-87f8f33da44a 5 2024-05-21T17:21:28.079 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:28.082+0000 7f8c3e7eb180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T17:21:28.158 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:28.158+0000 7f8c3e7eb180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T17:21:28.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.186+0000 7f5745f80640 10 monclient: get_monmap_and_config 2024-05-21T17:21:28.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.186+0000 7f5745f80640 10 monclient: build_initial_monmap 2024-05-21T17:21:28.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.186+0000 7f5745f80640 10 monclient: monmap: 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:21:28.189431+0000 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:21:28.189431+0000 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:28.186 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:28.187 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:28.187 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:28.187 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:28.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.186+0000 7f5745f80640 1 Processor -- start 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.186+0000 7f5745f80640 1 -- start start 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.186+0000 7f5745f80640 10 monclient: init 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: _add_conns ranks=[7,3,8,5,1,2,4,6,0] 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): picked mon.noname-i con 0x7f5740135b60 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): picked mon.noname-g con 0x7f574009dba0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): picked mon.noname-e con 0x7f574013f280 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574006c580 con 0x7f574009dba0 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574009cee0 con 0x7f5740135b60 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574009d1c0 con 0x7f574013f280 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): authenticate will time out at 943.256405s 2024-05-21T17:21:28.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573f7fe640 1 --1- >> v1:172.21.15.203:6792/0 conn(0x7f5740135b60 0x7f5740135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6792/0 says I am v1:172.21.15.203:41344/0 (socket says 172.21.15.203:41344) 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573f7fe640 1 -- 172.21.15.203:0/641656450 learned_addr learned my addr 172.21.15.203:0/641656450 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485108508 0 0) 0x7f574009cee0 con 0x7f5740135b60 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient(hunting): my global_id is 4138 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5728003530 con 0x7f5740135b60 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881290275 0 0) 0x7f574006c580 con 0x7f574009dba0 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient(hunting): my global_id is 4134 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f574009cee0 con 0x7f574009dba0 2024-05-21T17:21:28.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4021859503 0 0) 0x7f5728003530 con 0x7f5740135b60 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 >> v1:172.21.15.3:6793/0 conn(0x7f574013f280 legacy=0x7f5740141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 >> v1:172.21.15.203:6790/0 conn(0x7f574009dba0 legacy=0x7f574013db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 monclient: found mon.noname-i 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.203:6792/0 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 --> v1:172.21.15.203:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574006d120 con 0x7f5740135b60 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5734001840 con 0x7f5740135b60 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 5 monclient: authenticate success, global_id 4138 2024-05-21T17:21:28.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: dump: 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:28.193 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 <== mon.7 v1:172.21.15.203:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5734001a60 con 0x7f5740135b60 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 4 set_mon_vals no callback set 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 1 -- 172.21.15.203:0/641656450 <== mon.7 v1:172.21.15.203:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5734004bf0 con 0x7f5740135b60 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573e7fc640 10 monclient: dump: 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:28.195 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: shutdown 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- 172.21.15.203:0/641656450 >> v1:172.21.15.203:6792/0 conn(0x7f5740135b60 legacy=0x7f5740135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- 172.21.15.203:0/641656450 shutdown_connections 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- 172.21.15.203:0/641656450 >> 172.21.15.203:0/641656450 conn(0x7f57400a84e0 msgr2=0x7f57400a49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- 172.21.15.203:0/641656450 shutdown_connections 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- 172.21.15.203:0/641656450 wait complete. 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: build_initial_monmap 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: monmap: 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:28.196 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 Processor -- start 2024-05-21T17:21:28.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- start start 2024-05-21T17:21:28.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: init 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: _add_conns ranks=[3,8,0,1,5,7,4,2,6] 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): picked mon.noname-g con 0x7f574009dba0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): picked mon.noname-e con 0x7f5740135b60 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): picked mon.noname-a con 0x7f574013f280 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5740135f80 con 0x7f574013f280 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574013a710 con 0x7f574009dba0 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57401379b0 con 0x7f5740135b60 2024-05-21T17:21:28.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient(hunting): authenticate will time out at 943.259202s 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573f7fe640 1 --1- >> v1:172.21.15.203:6790/0 conn(0x7f574009dba0 0x7f5740136b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6790/0 says I am v1:172.21.15.203:59732/0 (socket says 172.21.15.203:59732) 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f573f7fe640 1 -- 172.21.15.203:0/3952064265 learned_addr learned my addr 172.21.15.203:0/3952064265 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2527702636 0 0) 0x7f574013a710 con 0x7f574009dba0 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient(hunting): my global_id is 4143 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f572c003530 con 0x7f574009dba0 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1826295056 0 0) 0x7f572c003530 con 0x7f574009dba0 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f574013a710 con 0x7f574009dba0 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5734004b30 con 0x7f574009dba0 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-05-21T17:21:28.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient(hunting): dump: 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:28.202 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1427601452 0 0) 0x7f574013a710 con 0x7f574009dba0 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 >> v1:172.21.15.3:6793/0 conn(0x7f5740135b60 legacy=0x7f57401372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 >> v1:172.21.15.3:6789/0 conn(0x7f574013f280 legacy=0x7f57401d5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 monclient: found mon.g 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 --> v1:172.21.15.203:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5740138ba0 con 0x7f574009dba0 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient: _finish_auth 0 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5727fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 5 monclient: authenticate success, global_id 4143 2024-05-21T17:21:28.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: _renew_subs 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.190+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 --> v1:172.21.15.203:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5740138720 con 0x7f574009dba0 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5745f80640 10 monclient: _renew_subs 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5745f80640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 --> v1:172.21.15.203:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f574013e290 con 0x7f574009dba0 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5734004eb0 con 0x7f574009dba0 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f57340053a0 con 0x7f574009dba0 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f573dffb640 4 set_mon_vals no callback set 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 10 monclient: dump: 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.204 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:21:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7f57340058c0 con 0x7f574009dba0 2024-05-21T17:21:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 2990448279 0 0) 0x7f5734006130 con 0x7f574009dba0 2024-05-21T17:21:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5745f80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5745f80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5745f80640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 --> v1:172.21.15.203:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5720005090 con 0x7f574009dba0 2024-05-21T17:21:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7f5734005bb0 con 0x7f574009dba0 2024-05-21T17:21:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.194+0000 7f5727fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:28.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.262+0000 7f5745f80640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "87412b84-97f1-4350-9034-87f8f33da44a", "id": 5}] 2024-05-21T17:21:28.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.262+0000 7f5745f80640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "87412b84-97f1-4350-9034-87f8f33da44a", "id": 5}] 2024-05-21T17:21:28.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.262+0000 7f5745f80640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:28.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.262+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 --> v1:172.21.15.203:6790/0 -- mon_command({"prefix": "osd new", "uuid": "87412b84-97f1-4350-9034-87f8f33da44a", "id": 5} v 0) -- 0x7f5720005380 con 0x7f574009dba0 2024-05-21T17:21:28.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5727fff640 1 -- 172.21.15.203:0/3952064265 <== mon.3 v1:172.21.15.203:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87412b84-97f1-4350-9034-87f8f33da44a", "id": 5}]=0 v8) ==== 112+0+2 (unknown 3702460048 0 1988412176) 0x7f5734005e50 con 0x7f574009dba0 2024-05-21T17:21:28.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5727fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "87412b84-97f1-4350-9034-87f8f33da44a", "id": 5}] 2024-05-21T17:21:28.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5727fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:28.269 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-05-21T17:21:28.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5745f80640 10 monclient: shutdown 2024-05-21T17:21:28.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5745f80640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:28.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 >> v1:172.21.15.203:6790/0 conn(0x7f574009dba0 legacy=0x7f5740136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 shutdown_connections 2024-05-21T17:21:28.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 >> 172.21.15.203:0/3952064265 conn(0x7f57400a84e0 msgr2=0x7f574013e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:28.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 shutdown_connections 2024-05-21T17:21:28.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-05-21T17:21:28.270+0000 7f5745f80640 1 -- 172.21.15.203:0/3952064265 wait complete. 2024-05-21T17:21:28.285 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-21T17:21:28.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-21T17:21:28.288 INFO:tasks.ceph.osd.0:Started 2024-05-21T17:21:28.288 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-21T17:21:28.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-21T17:21:28.291 INFO:tasks.ceph.osd.1:Started 2024-05-21T17:21:28.291 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-21T17:21:28.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-21T17:21:28.293 INFO:tasks.ceph.osd.2:Started 2024-05-21T17:21:28.293 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-21T17:21:28.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-21T17:21:28.314 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:28.314+0000 7f8c3e7eb180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T17:21:28.316 INFO:tasks.ceph.osd.3:Started 2024-05-21T17:21:28.316 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-21T17:21:28.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-21T17:21:28.318 INFO:tasks.ceph.osd.4:Started 2024-05-21T17:21:28.319 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-21T17:21:28.319 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-21T17:21:28.321 INFO:tasks.ceph.osd.5:Started 2024-05-21T17:21:28.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-21T17:21:28.385 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:28.386+0000 7f8c3e7eb180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T17:21:28.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient: get_monmap_and_config 2024-05-21T17:21:28.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient: build_initial_monmap 2024-05-21T17:21:28.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient: monmap: 2024-05-21T17:21:28.445 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:28.445 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.445 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:28.445562+0000 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:28.445562+0000 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:28.446 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:28.447 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 1 Processor -- start 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 1 -- start start 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient: init 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient: _add_conns ranks=[4,1,0,7,3,6,5,2,8] 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient(hunting): picked mon.noname-c con 0x7efe680a5bd0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient(hunting): picked mon.noname-f con 0x7efe680a66c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient(hunting): picked mon.noname-a con 0x7efe680a70c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe6809e480 con 0x7efe680a70c0 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe6809cce0 con 0x7efe680a66c0 2024-05-21T17:21:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe6809cfc0 con 0x7efe680a5bd0 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient(hunting): authenticate will time out at 955.602859s 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6d3ac640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7efe680a5bd0 0x7efe680a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:52902/0 (socket says 172.21.15.3:52902) 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6d3ac640 1 -- 172.21.15.3:0/1601885672 learned_addr learned my addr 172.21.15.3:0/1601885672 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4123585174 0 0) 0x7efe6809cfc0 con 0x7efe680a5bd0 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): my global_id is 4180 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe54003530 con 0x7efe680a5bd0 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2809776690 0 0) 0x7efe6809e480 con 0x7efe680a70c0 2024-05-21T17:21:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): my global_id is 4176 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe6809cfc0 con 0x7efe680a70c0 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440571964 0 0) 0x7efe6809cce0 con 0x7efe680a66c0 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient(hunting): my global_id is 4141 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe6809e480 con 0x7efe680a66c0 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1332371400 0 0) 0x7efe54003530 con 0x7efe680a5bd0 2024-05-21T17:21:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 >> v1:172.21.15.203:6789/0 conn(0x7efe680a66c0 legacy=0x7efe6813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 >> v1:172.21.15.3:6789/0 conn(0x7efe680a70c0 legacy=0x7efe68141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 monclient: found mon.noname-c 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe680742e0 con 0x7efe680a5bd0 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: _finish_auth 0 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efe64001840 con 0x7efe680a5bd0 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 5 monclient: authenticate success, global_id 4180 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:28.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:28.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: dump: 2024-05-21T17:21:28.456 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe640019e0 con 0x7efe680a5bd0 2024-05-21T17:21:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 1 -- 172.21.15.3:0/1601885672 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efe64004bf0 con 0x7efe680a5bd0 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe6f610640 4 set_mon_vals no callback set 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.440+0000 7efe5ffff640 10 monclient: dump: 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: shutdown 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- 172.21.15.3:0/1601885672 >> v1:172.21.15.3:6791/0 conn(0x7efe680a5bd0 legacy=0x7efe680a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- 172.21.15.3:0/1601885672 shutdown_connections 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- 172.21.15.3:0/1601885672 >> 172.21.15.3:0/1601885672 conn(0x7efe680ab7d0 msgr2=0x7efe680a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- 172.21.15.3:0/1601885672 shutdown_connections 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- 172.21.15.3:0/1601885672 wait complete. 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: build_initial_monmap 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: monmap: 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 Processor -- start 2024-05-21T17:21:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- start start 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: init 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: _add_conns ranks=[7,3,6,5,2,0,4,1,8] 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient(hunting): picked mon.noname-i con 0x7efe680a5bd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient(hunting): picked mon.noname-g con 0x7efe680a66c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient(hunting): picked mon.noname-d con 0x7efe680a70c0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe68140650 con 0x7efe680a66c0 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe68141300 con 0x7efe680a70c0 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe681cabd0 con 0x7efe680a5bd0 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient(hunting): authenticate will time out at 955.605865s 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6dbad640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7efe680a70c0 0x7efe681d58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:37056/0 (socket says 172.21.15.3:37056) 2024-05-21T17:21:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6dbad640 1 -- 172.21.15.3:0/3852379216 learned_addr learned my addr 172.21.15.3:0/3852379216 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2168107264 0 0) 0x7efe600015a0 con 0x7efe680a70c0 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): my global_id is 4146 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe4c003530 con 0x7efe680a70c0 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570571852 0 0) 0x7efe6809e480 con 0x7efe680a66c0 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): my global_id is 4188 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe600015a0 con 0x7efe680a66c0 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 998208759 0 0) 0x7efe4c003530 con 0x7efe680a70c0 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe6809e480 con 0x7efe680a70c0 2024-05-21T17:21:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3128339120 0 0) 0x7efe681cabd0 con 0x7efe680a5bd0 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): my global_id is 4165 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe4c003530 con 0x7efe680a5bd0 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efe60002010 con 0x7efe680a70c0 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): dump: 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1542296864 0 0) 0x7efe6809e480 con 0x7efe680a70c0 2024-05-21T17:21:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 >> v1:172.21.15.203:6792/0 conn(0x7efe680a5bd0 legacy=0x7efe681c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 >> v1:172.21.15.203:6790/0 conn(0x7efe680a66c0 legacy=0x7efe681ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 monclient: found mon.d 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe680a79f0 con 0x7efe680a70c0 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 5 monclient: authenticate success, global_id 4146 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: _renew_subs 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe6809e6a0 con 0x7efe680a70c0 2024-05-21T17:21:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: _renew_subs 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7efe68141ba0 con 0x7efe680a70c0 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe60003c30 con 0x7efe680a70c0 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efe600041e0 con 0x7efe680a70c0 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5f7fe640 4 set_mon_vals no callback set 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 10 monclient: dump: 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7efe60005b10 con 0x7efe680a70c0 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 2031640072 0 0) 0x7efe600071f0 con 0x7efe680a70c0 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.444+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.3:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe48005090 con 0x7efe680a70c0 2024-05-21T17:21:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.448+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7efe68141300 con 0x7efe680a70c0 2024-05-21T17:21:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.448+0000 7efe5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.448+0000 7efe5dffb640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 --> v1:172.21.15.3:6792/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe48005380 con 0x7efe680a70c0 2024-05-21T17:21:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe5dffb640 1 -- 172.21.15.3:0/3852379216 <== mon.6 v1:172.21.15.3:6792/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 2626281138) 0x7efe60006c00 con 0x7efe680a70c0 2024-05-21T17:21:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe5dffb640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:28.513 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:28.513 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","created":"2024-05-21T17:21:26.465014+0000","modified":"2024-05-21T17:21:28.263976+0000","last_up_change":"0.000000","last_in_change":"2024-05-21T17:21:28.263976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7dfaea0b-2159-4f6e-86a6-da6637052771","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"28d053ac-034d-43ed-a83e-9ee62a426b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9de90871-f66b-4a85-ac48-772541c5b23d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f8431d10-6318-458b-9a94-51fc47273f87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"028748a9-3ca4-4730-960e-8913c9504a9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"87412b84-97f1-4350-9034-87f8f33da44a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T17:21:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 10 monclient: shutdown 2024-05-21T17:21:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 >> v1:172.21.15.3:6792/0 conn(0x7efe680a70c0 legacy=0x7efe681d58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 shutdown_connections 2024-05-21T17:21:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 >> 172.21.15.3:0/3852379216 conn(0x7efe680ab7d0 msgr2=0x7efe68075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 shutdown_connections 2024-05-21T17:21:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:28.508+0000 7efe6f610640 1 -- 172.21.15.3:0/3852379216 wait complete. 2024-05-21T17:21:28.530 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-21T17:21:28.531 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-21T17:21:28.913 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-05-21T17:21:28.908+0000 7fa028eb97c0 -1 Falling back to public interface 2024-05-21T17:21:28.920 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-05-21T17:21:28.916+0000 7ff8ac0a37c0 -1 Falling back to public interface 2024-05-21T17:21:28.924 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-05-21T17:21:28.926+0000 7f4eda9fd7c0 -1 Falling back to public interface 2024-05-21T17:21:28.940 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-05-21T17:21:28.942+0000 7fbcd88877c0 -1 Falling back to public interface 2024-05-21T17:21:28.948 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-05-21T17:21:28.950+0000 7f718253c7c0 -1 Falling back to public interface 2024-05-21T17:21:28.949 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-05-21T17:21:28.944+0000 7f61138d27c0 -1 Falling back to public interface 2024-05-21T17:21:29.073 INFO:tasks.ceph.mgr.x.smithi203.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-21T17:21:29.073 INFO:tasks.ceph.mgr.x.smithi203.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-21T17:21:29.074 INFO:tasks.ceph.mgr.x.smithi203.stderr: from numpy import show_config as show_numpy_config 2024-05-21T17:21:29.135 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-21T17:21:29.136 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:29.138+0000 7f8c3e7eb180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T17:21:29.256 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient: get_monmap_and_config 2024-05-21T17:21:29.256 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient: build_initial_monmap 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient: monmap: 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:21:29.256745+0000 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:21:29.256745+0000 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:29.257 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:29.258 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:29.258 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:29.258 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:29.258 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:29.258 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 Processor -- start 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- start start 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient: init 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient: _add_conns ranks=[1,8,5,3,0,2,7,4,6] 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient(hunting): picked mon.noname-f con 0x7fb58c0a48d0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient(hunting): picked mon.noname-e con 0x7fb58c138e90 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient(hunting): picked mon.noname-h con 0x7fb58c13f3c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c0a8ab0 con 0x7fb58c0a48d0 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c0a2b00 con 0x7fb58c13f3c0 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c0a30c0 con 0x7fb58c138e90 2024-05-21T17:21:29.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient(hunting): authenticate will time out at 956.413995s 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb590ae5640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fb58c138e90 0x7fb58c13dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:53346/0 (socket says 172.21.15.3:53346) 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb590ae5640 1 -- 172.21.15.3:0/771439780 learned_addr learned my addr 172.21.15.3:0/771439780 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb591ae7640 1 --1- 172.21.15.3:0/771439780 >> v1:172.21.15.203:6791/0 conn(0x7fb58c13f3c0 0x7fb58c1417b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.3:43024/0 (socket says 172.21.15.3:43024) 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb5912e6640 1 --1- 172.21.15.3:0/771439780 >> v1:172.21.15.203:6789/0 conn(0x7fb58c0a48d0 0x7fb58c0a4cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.3:54186/0 (socket says 172.21.15.3:54186) 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339962683 0 0) 0x7fb58c0a30c0 con 0x7fb58c138e90 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): my global_id is 4184 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb578003530 con 0x7fb58c138e90 2024-05-21T17:21:29.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 77777530 0 0) 0x7fb58c0a8ab0 con 0x7fb58c0a48d0 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): my global_id is 4150 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb58c0a30c0 con 0x7fb58c0a48d0 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914526190 0 0) 0x7fb58c0a2b00 con 0x7fb58c13f3c0 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient(hunting): my global_id is 4217 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb58c0a8ab0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 655528682 0 0) 0x7fb578003530 con 0x7fb58c138e90 2024-05-21T17:21:29.266 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 >> v1:172.21.15.203:6791/0 conn(0x7fb58c13f3c0 legacy=0x7fb58c1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 >> v1:172.21.15.203:6789/0 conn(0x7fb58c0a48d0 legacy=0x7fb58c0a4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 monclient: found mon.noname-e 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c0a25e0 con 0x7fb58c138e90 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: _finish_auth 0 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb57c001840 con 0x7fb58c138e90 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:29.267 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: dump: 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:29.268 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 5 monclient: authenticate success, global_id 4184 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb57c001a60 con 0x7fb58c138e90 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 1 -- 172.21.15.3:0/771439780 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb57c004bf0 con 0x7fb58c138e90 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb583fff640 10 monclient: dump: 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:29.269 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:29.270 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 4 set_mon_vals no callback set 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 10 monclient: shutdown 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- 172.21.15.3:0/771439780 >> v1:172.21.15.3:6793/0 conn(0x7fb58c138e90 legacy=0x7fb58c13dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- 172.21.15.3:0/771439780 shutdown_connections 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- 172.21.15.3:0/771439780 >> 172.21.15.3:0/771439780 conn(0x7fb58c0ab740 msgr2=0x7fb58c0a7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- 172.21.15.3:0/771439780 shutdown_connections 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.252+0000 7fb59354a640 1 -- 172.21.15.3:0/771439780 wait complete. 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: build_initial_monmap 2024-05-21T17:21:29.271 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: monmap: 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:29.272 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 1 Processor -- start 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 1 -- start start 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: init 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: _add_conns ranks=[1,7,8,6,4,0,3,2,5] 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient(hunting): picked mon.noname-f con 0x7fb58c0a48d0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient(hunting): picked mon.noname-i con 0x7fb58c138e90 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient(hunting): picked mon.noname-e con 0x7fb58c13f3c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c09f6b0 con 0x7fb58c0a48d0 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c0a1de0 con 0x7fb58c138e90 2024-05-21T17:21:29.273 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c09d0d0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient(hunting): authenticate will time out at 956.417364s 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb591ae7640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fb58c13f3c0 0x7fb58c1d6210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:53348/0 (socket says 172.21.15.3:53348) 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb591ae7640 1 -- 172.21.15.3:0/3742932876 learned_addr learned my addr 172.21.15.3:0/3742932876 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3313680582 0 0) 0x7fb58c09d0d0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): my global_id is 4193 2024-05-21T17:21:29.274 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb570003530 con 0x7fb58c13f3c0 2024-05-21T17:21:29.277 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1008691573 0 0) 0x7fb58c09f6b0 con 0x7fb58c0a48d0 2024-05-21T17:21:29.277 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:29.277 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:29.277 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:29.277 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): my global_id is 4159 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb58c09d0d0 con 0x7fb58c0a48d0 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854958611 0 0) 0x7fb58c0a25e0 con 0x7fb58c138e90 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): my global_id is 4174 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb58c09f6b0 con 0x7fb58c138e90 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 65915211 0 0) 0x7fb570003530 con 0x7fb58c13f3c0 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb58c0a25e0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb588001d60 con 0x7fb58c13f3c0 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:29.278 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): dump: 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:29.279 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 286642608 0 0) 0x7fb58c0a25e0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 >> v1:172.21.15.203:6792/0 conn(0x7fb58c138e90 legacy=0x7fb58c1d2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 >> v1:172.21.15.203:6789/0 conn(0x7fb58c0a48d0 legacy=0x7fb58c0a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 monclient: found mon.e 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c0a72d0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:29.280 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 5 monclient: authenticate success, global_id 4193 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: _renew_subs 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb58c138210 con 0x7fb58c13f3c0 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: _renew_subs 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fb58c09deb0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb588002060 con 0x7fb58c13f3c0 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb588005160 con 0x7fb58c13f3c0 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb5837fe640 4 set_mon_vals no callback set 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:29.281 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 10 monclient: dump: 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:29.282 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:29.283 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:29.283 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:29.283 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 1) ==== 796+0+0 (unknown 1475100922 0 0) 0x7fb5880056c0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.283 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 2031640072 0 0) 0x7fb588006f40 con 0x7fb58c13f3c0 2024-05-21T17:21:29.283 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:29.283 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:29.283 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:29.283 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.256+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56c005090 con 0x7fb58c13f3c0 2024-05-21T17:21:29.284 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.260+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (unknown 1092875540 0 534071502) 0x7fb588006a60 con 0x7fb58c13f3c0 2024-05-21T17:21:29.284 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.260+0000 7fb581ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:29.284 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.260+0000 7fb581ffb640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:29.313 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:29.314+0000 7f8c3e7eb180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T17:21:29.323 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.316+0000 7fb59354a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:29.323 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.316+0000 7fb59354a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:29.323 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.316+0000 7fb59354a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:29.323 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.316+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb56c005380 con 0x7fb58c13f3c0 2024-05-21T17:21:29.324 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb581ffb640 1 -- 172.21.15.3:0/3742932876 <== mon.8 v1:172.21.15.3:6793/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 2626281138) 0x7fb588006cf0 con 0x7fb58c13f3c0 2024-05-21T17:21:29.324 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb581ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:29.324 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb581ffb640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:29.325 INFO:teuthology.misc.health.smithi003.stdout: 2024-05-21T17:21:29.325 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":8,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","created":"2024-05-21T17:21:26.465014+0000","modified":"2024-05-21T17:21:28.263976+0000","last_up_change":"0.000000","last_in_change":"2024-05-21T17:21:28.263976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7dfaea0b-2159-4f6e-86a6-da6637052771","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"28d053ac-034d-43ed-a83e-9ee62a426b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9de90871-f66b-4a85-ac48-772541c5b23d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f8431d10-6318-458b-9a94-51fc47273f87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"028748a9-3ca4-4730-960e-8913c9504a9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"87412b84-97f1-4350-9034-87f8f33da44a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T17:21:29.325 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb59354a640 10 monclient: shutdown 2024-05-21T17:21:29.325 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb59354a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 >> v1:172.21.15.3:6793/0 conn(0x7fb58c13f3c0 legacy=0x7fb58c1d6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 shutdown_connections 2024-05-21T17:21:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 >> 172.21.15.3:0/3742932876 conn(0x7fb58c0ab740 msgr2=0x7fb58c0a99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 shutdown_connections 2024-05-21T17:21:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:29.320+0000 7fb59354a640 1 -- 172.21.15.3:0/3742932876 wait complete. 2024-05-21T17:21:29.340 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-05-21T17:21:29.497 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:29.498+0000 7f8c3e7eb180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T17:21:29.560 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:29.562+0000 7f8c3e7eb180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T17:21:29.694 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:29.694+0000 7f8c3e7eb180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T17:21:30.338 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:30.338+0000 7f8c3e7eb180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T17:21:30.416 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:30.418+0000 7f8c3e7eb180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T17:21:30.609 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:30.610+0000 7f8c3e7eb180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T17:21:30.682 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:30.682+0000 7f8c3e7eb180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T17:21:30.964 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:30.966+0000 7f8c3e7eb180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T17:21:31.025 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:31.026+0000 7f8c3e7eb180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T17:21:32.070 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:32.070+0000 7f8be672e640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-05-21T17:21:32.070 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:32.070+0000 7f8be672e640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-05-21T17:21:32.070 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:32.070+0000 7f8be672e640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-05-21T17:21:32.070 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:32.070+0000 7f8be672e640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-21T17:21:32.071 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:32.074+0000 7f8be672e640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-05-21T17:21:32.073 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:32.074+0000 7f8be672e640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-21T17:21:32.074 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:32.074+0000 7f8be672e640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-05-21T17:21:32.077 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-05-21T17:21:32.078+0000 7f8be672e640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-05-21T17:21:32.693 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-05-21T17:21:32.688+0000 7ff8ac0a37c0 -1 osd.2 0 log_to_monitors true 2024-05-21T17:21:32.698 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-05-21T17:21:32.692+0000 7fa028eb97c0 -1 osd.0 0 log_to_monitors true 2024-05-21T17:21:32.716 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-05-21T17:21:32.718+0000 7fbcd88877c0 -1 osd.3 0 log_to_monitors true 2024-05-21T17:21:32.716 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-05-21T17:21:32.718+0000 7f4eda9fd7c0 -1 osd.4 0 log_to_monitors true 2024-05-21T17:21:32.779 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-05-21T17:21:32.772+0000 7f61138d27c0 -1 osd.1 0 log_to_monitors true 2024-05-21T17:21:32.814 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-05-21T17:21:32.814+0000 7f718253c7c0 -1 osd.5 0 log_to_monitors true 2024-05-21T17:21:34.147 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-05-21T17:21:34.150+0000 7fbcd4830640 -1 osd.3 0 waiting for initial osdmap 2024-05-21T17:21:34.152 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-05-21T17:21:34.148+0000 7ff8a804c640 -1 osd.2 0 waiting for initial osdmap 2024-05-21T17:21:34.153 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-05-21T17:21:34.154+0000 7fbcd066c640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T17:21:34.159 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-05-21T17:21:34.152+0000 7ff8a3e88640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T17:21:34.160 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-05-21T17:21:34.162+0000 7f717e4e5640 -1 osd.5 0 waiting for initial osdmap 2024-05-21T17:21:34.167 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-05-21T17:21:34.170+0000 7f717a321640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T17:21:34.807 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-05-21T17:21:34.810+0000 7f4ed69a6640 -1 osd.4 0 waiting for initial osdmap 2024-05-21T17:21:34.814 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-05-21T17:21:34.814+0000 7f4ece7da640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T17:21:34.966 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-05-21T17:21:34.960+0000 7f611008d640 -1 osd.1 0 waiting for initial osdmap 2024-05-21T17:21:34.969 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-05-21T17:21:34.964+0000 7f61076af640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T17:21:34.976 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-05-21T17:21:34.968+0000 7fa024e62640 -1 osd.0 0 waiting for initial osdmap 2024-05-21T17:21:34.979 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-05-21T17:21:34.976+0000 7fa01cc96640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T17:21:35.946 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-21T17:21:36.071 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.064+0000 7f373e401640 10 monclient: get_monmap_and_config 2024-05-21T17:21:36.071 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.064+0000 7f373e401640 10 monclient: build_initial_monmap 2024-05-21T17:21:36.071 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.064+0000 7f373e401640 10 monclient: monmap: 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:21:36.071436+0000 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:21:36.071436+0000 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:36.072 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:36.073 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:36.073 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:36.073 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:36.073 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:36.073 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:36.073 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:36.079 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 Processor -- start 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- start start 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient: init 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient: _add_conns ranks=[3,7,5,8,0,4,1,6,2] 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient(hunting): picked mon.noname-g con 0x7f37380a4ac0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient(hunting): picked mon.noname-i con 0x7f37380a4ea0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient(hunting): picked mon.noname-h con 0x7f37380a5280 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37380a2ac0 con 0x7f37380a4ac0 2024-05-21T17:21:36.080 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37380a2da0 con 0x7f37380a5280 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3738074230 con 0x7f37380a4ea0 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient(hunting): authenticate will time out at 963.229021s 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3737fff640 1 --1- >> v1:172.21.15.203:6790/0 conn(0x7f37380a4ac0 0x7f3738138ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6790/0 says I am v1:172.21.15.3:40654/0 (socket says 172.21.15.3:40654) 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3737fff640 1 -- 172.21.15.3:0/3658242350 learned_addr learned my addr 172.21.15.3:0/3658242350 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f37377fe640 1 --1- >> v1:172.21.15.203:6792/0 conn(0x7f37380a4ea0 0x7f373813dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6792/0 says I am v1:172.21.15.3:38710/0 (socket says 172.21.15.3:38710) 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373c99e640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f37380a5280 0x7f3738141460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.3:58128/0 (socket says 172.21.15.3:58128) 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659439162 0 0) 0x7f37380a2ac0 con 0x7f37380a4ac0 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.081 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): my global_id is 4206 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3720003530 con 0x7f37380a4ac0 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 869490403 0 0) 0x7f37380a2da0 con 0x7f37380a5280 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): my global_id is 4271 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37380a2ac0 con 0x7f37380a5280 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198367067 0 0) 0x7f3738074230 con 0x7f37380a4ea0 2024-05-21T17:21:36.082 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient(hunting): my global_id is 4228 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37380a2da0 con 0x7f37380a4ea0 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1285917528 0 0) 0x7f3720003530 con 0x7f37380a4ac0 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 >> v1:172.21.15.203:6792/0 conn(0x7f37380a4ea0 legacy=0x7f373813dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 >> v1:172.21.15.203:6791/0 conn(0x7f37380a5280 legacy=0x7f3738141460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 monclient: found mon.noname-g 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:36.083 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.203:6790/0 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 --> v1:172.21.15.203:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3738145bb0 con 0x7f37380a4ac0 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3728001bc0 con 0x7f37380a4ac0 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: dump: 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.084 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.085 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 5 monclient: authenticate success, global_id 4206 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 <== mon.3 v1:172.21.15.203:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3728001d60 con 0x7f37380a4ac0 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 1 -- 172.21.15.3:0/3658242350 <== mon.3 v1:172.21.15.203:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3728004f70 con 0x7f37380a4ac0 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f3736ffd640 10 monclient: dump: 2024-05-21T17:21:36.086 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.087 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 4 set_mon_vals no callback set 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 10 monclient: shutdown 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- 172.21.15.3:0/3658242350 >> v1:172.21.15.203:6790/0 conn(0x7f37380a4ac0 legacy=0x7f3738138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- 172.21.15.3:0/3658242350 shutdown_connections 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- 172.21.15.3:0/3658242350 >> 172.21.15.3:0/3658242350 conn(0x7f37380ab7f0 msgr2=0x7f37380a7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:36.088 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- 172.21.15.3:0/3658242350 shutdown_connections 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.068+0000 7f373e401640 1 -- 172.21.15.3:0/3658242350 wait complete. 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: build_initial_monmap 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: monmap: 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:36.089 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 1 Processor -- start 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 1 -- start start 2024-05-21T17:21:36.090 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: init 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: _add_conns ranks=[0,4,1,3,2,7,5,8,6] 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient(hunting): picked mon.noname-a con 0x7f37380a4ac0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient(hunting): picked mon.noname-c con 0x7f37380a4ea0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient(hunting): picked mon.noname-f con 0x7f37380a5280 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373813e510 con 0x7f37380a4ac0 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37380a7380 con 0x7f37380a5280 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37380a1e50 con 0x7f37380a4ea0 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient(hunting): authenticate will time out at 963.233471s 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3737fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f37380a4ac0 0x7f37381cdd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34808/0 (socket says 172.21.15.3:34808) 2024-05-21T17:21:36.091 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3737fff640 1 -- 172.21.15.3:0/585080952 learned_addr learned my addr 172.21.15.3:0/585080952 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1684852975 0 0) 0x7f37380a1e50 con 0x7f37380a4ea0 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): my global_id is 4225 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3724003530 con 0x7f37380a4ea0 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761345115 0 0) 0x7f373813e510 con 0x7f37380a4ac0 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): my global_id is 4221 2024-05-21T17:21:36.092 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37380a1e50 con 0x7f37380a4ac0 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3870907492 0 0) 0x7f37380a7380 con 0x7f37380a5280 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): my global_id is 4195 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f373813e510 con 0x7f37380a5280 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3097395530 0 0) 0x7f37380a1e50 con 0x7f37380a4ac0 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37380a7380 con 0x7f37380a4ac0 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1647097294 0 0) 0x7f3724003530 con 0x7f37380a4ea0 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37380a1e50 con 0x7f37380a4ea0 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3728001b80 con 0x7f37380a4ac0 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:36.093 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): dump: 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.094 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f372c0019a0 con 0x7f37380a4ea0 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:36.095 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient(hunting): dump: 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.096 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1418312888 0 0) 0x7f37380a7380 con 0x7f37380a4ac0 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 >> v1:172.21.15.3:6791/0 conn(0x7f37380a4ea0 legacy=0x7f37381d2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 >> v1:172.21.15.203:6789/0 conn(0x7f37380a5280 legacy=0x7f37381d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient: _finish_hunting 0 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 monclient: found mon.a 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:36.097 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373813df50 con 0x7f37380a4ac0 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient: _finish_auth 0 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient: _check_auth_tickets 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 5 monclient: authenticate success, global_id 4221 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: _renew_subs 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f373813e230 con 0x7f37380a4ac0 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: _renew_subs 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f373813b590 con 0x7f37380a4ac0 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3728001d20 con 0x7f37380a4ac0 2024-05-21T17:21:36.098 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3728004f80 con 0x7f37380a4ac0 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f37367fc640 4 set_mon_vals no callback set 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 10 monclient: dump: 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.099 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr: 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:36.100 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3718005090 con 0x7f37380a4ac0 2024-05-21T17:21:36.101 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7f3728010560 con 0x7f37380a4ac0 2024-05-21T17:21:36.101 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.072+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(11..11 src has 1..11) ==== 2767+0+0 (unknown 3951629679 0 0) 0x7f3728045bc0 con 0x7f37380a4ac0 2024-05-21T17:21:36.101 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.076+0000 7f37377fe640 10 monclient: get_auth_request con 0x7f37240393c0 auth_method 0 2024-05-21T17:21:36.101 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.080+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f3728016050 con 0x7f37380a4ac0 2024-05-21T17:21:36.101 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.080+0000 7f3734ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:36.101 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.080+0000 7f3734ff9640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:36.261 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.256+0000 7f373e401640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:36.261 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.256+0000 7f373e401640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:36.261 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.256+0000 7f373e401640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:36.261 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.256+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3718005380 con 0x7f37380a4ac0 2024-05-21T17:21:36.261 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.256+0000 7f3734ff9640 1 -- 172.21.15.3:0/585080952 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7155 (unknown 4250787923 0 1317197240) 0x7f3728019ca0 con 0x7f37380a4ac0 2024-05-21T17:21:36.262 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.256+0000 7f3734ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:36.262 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.256+0000 7f3734ff9640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:36.262 INFO:teuthology.misc.health.smithi003.stdout: 2024-05-21T17:21:36.262 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":12,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","created":"2024-05-21T17:21:26.465014+0000","modified":"2024-05-21T17:21:36.173809+0000","last_up_change":"2024-05-21T17:21:35.154389+0000","last_in_change":"2024-05-21T17:21:28.263976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7dfaea0b-2159-4f6e-86a6-da6637052771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6800","nonce":857120552}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6801","nonce":857120552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6803","nonce":857120552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6802","nonce":857120552}]},"public_addr":"172.21.15.3:6800/857120552","cluster_addr":"172.21.15.3:6801/857120552","heartbeat_back_addr":"172.21.15.3:6803/857120552","heartbeat_front_addr":"172.21.15.3:6802/857120552","state":["exists","up"]},{"osd":1,"uuid":"28d053ac-034d-43ed-a83e-9ee62a426b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6808","nonce":4137544047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6809","nonce":4137544047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6811","nonce":4137544047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6810","nonce":4137544047}]},"public_addr":"172.21.15.3:6808/4137544047","cluster_addr":"172.21.15.3:6809/4137544047","heartbeat_back_addr":"172.21.15.3:6811/4137544047","heartbeat_front_addr":"172.21.15.3:6810/4137544047","state":["exists","up"]},{"osd":2,"uuid":"9de90871-f66b-4a85-ac48-772541c5b23d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6804","nonce":517111994}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6805","nonce":517111994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6807","nonce":517111994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6806","nonce":517111994}]},"public_addr":"172.21.15.3:6804/517111994","cluster_addr":"172.21.15.3:6805/517111994","heartbeat_back_addr":"172.21.15.3:6807/517111994","heartbeat_front_addr":"172.21.15.3:6806/517111994","state":["exists","up"]},{"osd":3,"uuid":"f8431d10-6318-458b-9a94-51fc47273f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":1468009857}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1468009857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1468009857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":1468009857}]},"public_addr":"172.21.15.203:6804/1468009857","cluster_addr":"172.21.15.203:6805/1468009857","heartbeat_back_addr":"172.21.15.203:6807/1468009857","heartbeat_front_addr":"172.21.15.203:6806/1468009857","state":["exists","up"]},{"osd":4,"uuid":"028748a9-3ca4-4730-960e-8913c9504a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":4020495055}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":4020495055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":4020495055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":4020495055}]},"public_addr":"172.21.15.203:6800/4020495055","cluster_addr":"172.21.15.203:6801/4020495055","heartbeat_back_addr":"172.21.15.203:6803/4020495055","heartbeat_front_addr":"172.21.15.203:6802/4020495055","state":["exists","up"]},{"osd":5,"uuid":"87412b84-97f1-4350-9034-87f8f33da44a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":2732955435}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":2732955435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":2732955435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":2732955435}]},"public_addr":"172.21.15.203:6808/2732955435","cluster_addr":"172.21.15.203:6809/2732955435","heartbeat_back_addr":"172.21.15.203:6811/2732955435","heartbeat_front_addr":"172.21.15.203:6810/2732955435","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.740995+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.776640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.734901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.745181+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.676894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.827672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T17:21:36.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.260+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 >> v1:172.21.15.203:6812/343827626 conn(0x7f37240393c0 legacy=0x7f372403b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.260+0000 7f373e401640 10 monclient: shutdown 2024-05-21T17:21:36.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.260+0000 7f373e401640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:36.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.260+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 >> v1:172.21.15.3:6789/0 conn(0x7f37380a4ac0 legacy=0x7f37381cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.260+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 shutdown_connections 2024-05-21T17:21:36.264 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.260+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 >> 172.21.15.3:0/585080952 conn(0x7f37380ab7f0 msgr2=0x7f37380aa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:36.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.260+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 shutdown_connections 2024-05-21T17:21:36.265 INFO:teuthology.misc.health.smithi003.stderr:2024-05-21T17:21:36.260+0000 7f373e401640 1 -- 172.21.15.3:0/585080952 wait complete. 2024-05-21T17:21:36.279 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-05-21T17:21:36.279 INFO:tasks.ceph:Creating RBD pool 2024-05-21T17:21:36.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.392+0000 7efc35741640 10 monclient: get_monmap_and_config 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.392+0000 7efc35741640 10 monclient: build_initial_monmap 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.392+0000 7efc35741640 10 monclient: monmap: 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:36.400066+0000 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:36.400066+0000 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.400 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:36.401 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 Processor -- start 2024-05-21T17:21:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- start start 2024-05-21T17:21:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient: init 2024-05-21T17:21:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient: _add_conns ranks=[2,3,4,0,7,6,8,1,5] 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient(hunting): picked mon.noname-b con 0x7efc300a2e20 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient(hunting): picked mon.noname-g con 0x7efc300a1680 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient(hunting): picked mon.noname-c con 0x7efc3013b0c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc300a0150 con 0x7efc300a2e20 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc30134d50 con 0x7efc300a1680 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc300a03c0 con 0x7efc3013b0c0 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient(hunting): authenticate will time out at 963.557987s 2024-05-21T17:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2effd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7efc300a2e20 0x7efc30134ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47082/0 (socket says 172.21.15.3:47082) 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2effd640 1 -- 172.21.15.3:0/1121918453 learned_addr learned my addr 172.21.15.3:0/1121918453 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1675897879 0 0) 0x7efc300a0150 con 0x7efc300a2e20 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): my global_id is 4178 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc1c003450 con 0x7efc300a2e20 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590025166 0 0) 0x7efc300a03c0 con 0x7efc3013b0c0 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): my global_id is 4234 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc300a0150 con 0x7efc3013b0c0 2024-05-21T17:21:36.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007657618 0 0) 0x7efc30134d50 con 0x7efc300a1680 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient(hunting): my global_id is 4215 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc300a03c0 con 0x7efc300a1680 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1255279231 0 0) 0x7efc1c003450 con 0x7efc300a2e20 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 >> v1:172.21.15.3:6791/0 conn(0x7efc3013b0c0 legacy=0x7efc3013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 >> v1:172.21.15.203:6790/0 conn(0x7efc300a1680 legacy=0x7efc30135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 monclient: found mon.noname-b 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc300a06a0 con 0x7efc300a2e20 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc20001840 con 0x7efc300a2e20 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: dump: 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.411 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 5 monclient: authenticate success, global_id 4178 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efc20001a60 con 0x7efc300a2e20 2024-05-21T17:21:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 1 -- 172.21.15.3:0/1121918453 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc20001d50 con 0x7efc300a2e20 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 4 set_mon_vals no callback set 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc2e7fc640 10 monclient: dump: 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 10 monclient: shutdown 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- 172.21.15.3:0/1121918453 >> v1:172.21.15.3:6790/0 conn(0x7efc300a2e20 legacy=0x7efc30134ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- 172.21.15.3:0/1121918453 shutdown_connections 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- 172.21.15.3:0/1121918453 >> 172.21.15.3:0/1121918453 conn(0x7efc300a3f10 msgr2=0x7efc300a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- 172.21.15.3:0/1121918453 shutdown_connections 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.396+0000 7efc35741640 1 -- 172.21.15.3:0/1121918453 wait complete. 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: build_initial_monmap 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: monmap: 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 1 Processor -- start 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 1 -- start start 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: init 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: _add_conns ranks=[3,4,5,0,8,7,6,1,2] 2024-05-21T17:21:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient(hunting): picked mon.noname-g con 0x7efc300a1680 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient(hunting): picked mon.noname-c con 0x7efc300a2e20 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient(hunting): picked mon.noname-h con 0x7efc3013b0c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc30099cd0 con 0x7efc300a1680 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc300a3870 con 0x7efc300a2e20 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc3012c920 con 0x7efc3013b0c0 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient(hunting): authenticate will time out at 963.561402s 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc27fff640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7efc300a2e20 0x7efc301d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47488/0 (socket says 172.21.15.3:47488) 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc27fff640 1 -- 172.21.15.3:0/3006504858 learned_addr learned my addr 172.21.15.3:0/3006504858 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2984227307 0 0) 0x7efc300a3870 con 0x7efc300a2e20 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): my global_id is 4243 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc10003530 con 0x7efc300a2e20 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3009193254 0 0) 0x7efc10003530 con 0x7efc300a2e20 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc300a3870 con 0x7efc300a2e20 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc18001b20 con 0x7efc300a2e20 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): dump: 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2079982393 0 0) 0x7efc30099cd0 con 0x7efc300a1680 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): my global_id is 4224 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc10003530 con 0x7efc300a1680 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547611785 0 0) 0x7efc3012c920 con 0x7efc3013b0c0 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient(hunting): my global_id is 4280 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc30099cd0 con 0x7efc3013b0c0 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2770111852 0 0) 0x7efc300a3870 con 0x7efc300a2e20 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 >> v1:172.21.15.203:6791/0 conn(0x7efc3013b0c0 legacy=0x7efc301d5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 >> v1:172.21.15.203:6790/0 conn(0x7efc300a1680 legacy=0x7efc3012e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 monclient: found mon.c 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3012cc00 con 0x7efc300a2e20 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 5 monclient: authenticate success, global_id 4243 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: _renew_subs 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efc3009a660 con 0x7efc300a2e20 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: _renew_subs 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:36.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7efc3012dbd0 con 0x7efc300a2e20 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efc180038d0 con 0x7efc300a2e20 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc18003e80 con 0x7efc300a2e20 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc2dffb640 4 set_mon_vals no callback set 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 10 monclient: dump: 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7efc180040a0 con 0x7efc300a2e20 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:36.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc14005090 con 0x7efc300a2e20 2024-05-21T17:21:36.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(12..12 src has 1..12) ==== 2767+0+0 (unknown 2747574643 0 0) 0x7efc18045910 con 0x7efc300a2e20 2024-05-21T17:21:36.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.400+0000 7efc2effd640 10 monclient: get_auth_request con 0x7efc10038e50 auth_method 0 2024-05-21T17:21:36.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.408+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7efc18015d40 con 0x7efc300a2e20 2024-05-21T17:21:36.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.408+0000 7efc277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:36.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.408+0000 7efc277fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:36.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.568+0000 7efc35741640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-21T17:21:36.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.568+0000 7efc35741640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-21T17:21:36.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.568+0000 7efc35741640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:36.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:36.568+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7efc14005c50 con 0x7efc300a2e20 2024-05-21T17:21:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.200+0000 7efc277fe640 1 -- 172.21.15.3:0/3006504858 <== mon.4 v1:172.21.15.3:6791/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (unknown 2984082091 0 0) 0x7efc180199f0 con 0x7efc300a2e20 2024-05-21T17:21:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.200+0000 7efc277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-21T17:21:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.200+0000 7efc277fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-05-21T17:21:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'rbd' created 2024-05-21T17:21:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.204+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 >> v1:172.21.15.203:6812/343827626 conn(0x7efc10038e50 legacy=0x7efc1003b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.204+0000 7efc35741640 10 monclient: shutdown 2024-05-21T17:21:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.204+0000 7efc35741640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.204+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 >> v1:172.21.15.3:6791/0 conn(0x7efc300a2e20 legacy=0x7efc301d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.204+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 shutdown_connections 2024-05-21T17:21:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.204+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 >> 172.21.15.3:0/3006504858 conn(0x7efc300a3f10 msgr2=0x7efc3009f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.204+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 shutdown_connections 2024-05-21T17:21:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.204+0000 7efc35741640 1 -- 172.21.15.3:0/3006504858 wait complete. 2024-05-21T17:21:37.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.352+0000 7efca456e640 10 monclient: get_monmap_and_config 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.352+0000 7efca456e640 10 monclient: build_initial_monmap 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.352+0000 7efca456e640 10 monclient: monmap: 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:37.359088+0000 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:37.359088+0000 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:37.360 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:37.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 Processor -- start 2024-05-21T17:21:37.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- start start 2024-05-21T17:21:37.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient: init 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient: _add_conns ranks=[5,7,1,3,4,2,6,8,0] 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient(hunting): picked mon.noname-h con 0x7efc9c0a38f0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient(hunting): picked mon.noname-i con 0x7efc9c0a43e0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient(hunting): picked mon.noname-f con 0x7efc9c09dc80 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c09e760 con 0x7efc9c09dc80 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c06d140 con 0x7efc9c0a38f0 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c06c7f0 con 0x7efc9c0a43e0 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient(hunting): authenticate will time out at 964.516964s 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca230a640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7efc9c0a38f0 0x7efc9c0a3cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.3:58144/0 (socket says 172.21.15.3:58144) 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1b09640 1 --1- >> v1:172.21.15.203:6792/0 conn(0x7efc9c0a43e0 0x7efc9c13de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6792/0 says I am v1:172.21.15.3:38716/0 (socket says 172.21.15.3:38716) 2024-05-21T17:21:37.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca230a640 1 -- 172.21.15.3:0/406611758 learned_addr learned my addr 172.21.15.3:0/406611758 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 58994227 0 0) 0x7efc9c09e760 con 0x7efc9c09dc80 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _negotiate 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): my global_id is 4204 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc88003530 con 0x7efc9c09dc80 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098945802 0 0) 0x7efc9c06d140 con 0x7efc9c0a38f0 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _negotiate 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:37.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): my global_id is 4289 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c09e760 con 0x7efc9c0a38f0 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288676950 0 0) 0x7efc9c06c7f0 con 0x7efc9c0a43e0 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _negotiate 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient(hunting): my global_id is 4237 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c06d140 con 0x7efc9c0a43e0 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2266749849 0 0) 0x7efc88003530 con 0x7efc9c09dc80 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 >> v1:172.21.15.203:6792/0 conn(0x7efc9c0a43e0 legacy=0x7efc9c13de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 >> v1:172.21.15.203:6791/0 conn(0x7efc9c0a38f0 legacy=0x7efc9c0a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: _finish_hunting 0 2024-05-21T17:21:37.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 monclient: found mon.noname-f 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.203:6789/0 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9c09cfc0 con 0x7efc9c09dc80 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: _finish_auth 0 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: _check_auth_tickets 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc98001840 con 0x7efc9c09dc80 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: dump: 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:37.371 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:37.372 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 5 monclient: authenticate success, global_id 4204 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efc98001a60 con 0x7efc9c09dc80 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 1 -- 172.21.15.3:0/406611758 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc98003960 con 0x7efc9c09dc80 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca1308640 10 monclient: dump: 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.373 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 4 set_mon_vals no callback set 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient: shutdown 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- 172.21.15.3:0/406611758 >> v1:172.21.15.203:6789/0 conn(0x7efc9c09dc80 legacy=0x7efc9c141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- 172.21.15.3:0/406611758 shutdown_connections 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- 172.21.15.3:0/406611758 >> 172.21.15.3:0/406611758 conn(0x7efc9c0a84e0 msgr2=0x7efc9c0a49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- 172.21.15.3:0/406611758 shutdown_connections 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 1 -- 172.21.15.3:0/406611758 wait complete. 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient: build_initial_monmap 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.356+0000 7efca456e640 10 monclient: monmap: 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:37.375 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:37.376 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 1 Processor -- start 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 1 -- start start 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: init 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: _add_conns ranks=[5,4,0,2,3,7,1,8,6] 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient(hunting): picked mon.noname-h con 0x7efc9c09dc80 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient(hunting): picked mon.noname-c con 0x7efc9c0a38f0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient(hunting): picked mon.noname-a con 0x7efc9c0a43e0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c126670 con 0x7efc9c0a43e0 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c126250 con 0x7efc9c0a38f0 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c09e980 con 0x7efc9c09dc80 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:37.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient(hunting): authenticate will time out at 964.520726s 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca1b09640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7efc9c0a38f0 0x7efc9c1229f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47494/0 (socket says 172.21.15.3:47494) 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca2b0b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efc9c0a43e0 0x7efc9c123100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34818/0 (socket says 172.21.15.3:34818) 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca1b09640 1 -- 172.21.15.3:0/4081653956 learned_addr learned my addr 172.21.15.3:0/4081653956 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130530007 0 0) 0x7efc9c126250 con 0x7efc9c0a38f0 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): my global_id is 4252 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc80003530 con 0x7efc9c0a38f0 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 169452684 0 0) 0x7efc9c09cfc0 con 0x7efc9c0a43e0 2024-05-21T17:21:37.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): my global_id is 4230 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c126250 con 0x7efc9c0a43e0 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1195789399 0 0) 0x7efc80003530 con 0x7efc9c0a38f0 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc9c09cfc0 con 0x7efc9c0a38f0 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc8c001ac0 con 0x7efc9c0a38f0 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:37.379 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 550715773 0 0) 0x7efc9c126250 con 0x7efc9c0a43e0 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc80003530 con 0x7efc9c0a43e0 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2424554598 0 0) 0x7efc9c09e980 con 0x7efc9c09dc80 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): my global_id is 4298 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c126250 con 0x7efc9c09dc80 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc98001800 con 0x7efc9c0a43e0 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:37.381 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:37.382 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1521771491 0 0) 0x7efc9c09cfc0 con 0x7efc9c0a38f0 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 >> v1:172.21.15.203:6791/0 conn(0x7efc9c09dc80 legacy=0x7efc9c1222e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 >> v1:172.21.15.3:6789/0 conn(0x7efc9c0a43e0 legacy=0x7efc9c123100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 monclient: found mon.c 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9c1268f0 con 0x7efc9c0a38f0 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 5 monclient: authenticate success, global_id 4252 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: _renew_subs 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efc9c124810 con 0x7efc9c0a38f0 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: _renew_subs 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7efc9c125f70 con 0x7efc9c0a38f0 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efc8c004aa0 con 0x7efc9c0a38f0 2024-05-21T17:21:37.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efc8c004f90 con 0x7efc9c0a38f0 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca0b07640 4 set_mon_vals no callback set 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 10 monclient: dump: 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:37.385 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7efc8c0100e0 con 0x7efc9c0a38f0 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc84005090 con 0x7efc9c0a38f0 2024-05-21T17:21:37.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(13..13 src has 1..13) ==== 3484+0+0 (unknown 2964303986 0 0) 0x7efc8c045ae0 con 0x7efc9c0a38f0 2024-05-21T17:21:37.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.360+0000 7efca230a640 10 monclient: get_auth_request con 0x7efc80039300 auth_method 0 2024-05-21T17:21:37.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.364+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7efc8c015d20 con 0x7efc9c0a38f0 2024-05-21T17:21:37.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.364+0000 7efc92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:37.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.364+0000 7efc92ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.532+0000 7efca456e640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-21T17:21:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.532+0000 7efca456e640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-21T17:21:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.532+0000 7efca456e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:37.532+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7efc84005380 con 0x7efc9c0a38f0 2024-05-21T17:21:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-21T17:21:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.204+0000 7efc92ffd640 1 -- 172.21.15.3:0/4081653956 <== mon.4 v1:172.21.15.3:6791/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 173+0+0 (unknown 3289125425 0 0) 0x7efc8c015620 con 0x7efc9c0a38f0 2024-05-21T17:21:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.204+0000 7efc92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-21T17:21:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.204+0000 7efc92ffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-05-21T17:21:38.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.212+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 >> v1:172.21.15.203:6812/343827626 conn(0x7efc80039300 legacy=0x7efc8003b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.212+0000 7efca456e640 10 monclient: shutdown 2024-05-21T17:21:38.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.212+0000 7efca456e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:38.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.212+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 >> v1:172.21.15.3:6791/0 conn(0x7efc9c0a38f0 legacy=0x7efc9c1229f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.212+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 shutdown_connections 2024-05-21T17:21:38.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.212+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 >> 172.21.15.3:0/4081653956 conn(0x7efc9c0a84e0 msgr2=0x7efc9c06bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:38.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.212+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 shutdown_connections 2024-05-21T17:21:38.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.212+0000 7efca456e640 1 -- 172.21.15.3:0/4081653956 wait complete. 2024-05-21T17:21:38.240 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-21T17:21:38.241 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-21T17:21:38.241 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-21T17:21:38.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-21T17:21:38.242 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-21T17:21:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.356+0000 7f226c1b5640 10 monclient: get_monmap_and_config 2024-05-21T17:21:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.356+0000 7f226c1b5640 10 monclient: build_initial_monmap 2024-05-21T17:21:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.356+0000 7f226c1b5640 10 monclient: monmap: 2024-05-21T17:21:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:38.363276+0000 2024-05-21T17:21:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:38.363276+0000 2024-05-21T17:21:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:38.365 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.356+0000 7f226c1b5640 1 Processor -- start 2024-05-21T17:21:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- start start 2024-05-21T17:21:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient: init 2024-05-21T17:21:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient: _add_conns ranks=[3,7,5,4,8,1,0,6,2] 2024-05-21T17:21:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient(hunting): picked mon.noname-g con 0x7f22641359b0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient(hunting): picked mon.noname-i con 0x7f22640a1620 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient(hunting): picked mon.noname-h con 0x7f2264135560 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22640a0290 con 0x7f22641359b0 2024-05-21T17:21:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22640698e0 con 0x7f2264135560 2024-05-21T17:21:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22640a0960 con 0x7f22640a1620 2024-05-21T17:21:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient(hunting): authenticate will time out at 965.520743s 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2269750640 1 --1- >> v1:172.21.15.203:6792/0 conn(0x7f22640a1620 0x7f226413a880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6792/0 says I am v1:172.21.15.3:38722/0 (socket says 172.21.15.3:38722) 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2269750640 1 -- 172.21.15.3:0/1554278841 learned_addr learned my addr 172.21.15.3:0/1554278841 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075547659 0 0) 0x7f22640a0290 con 0x7f22641359b0 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): my global_id is 4233 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2250003530 con 0x7f22641359b0 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029314586 0 0) 0x7f22640a0960 con 0x7f22640a1620 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): my global_id is 4246 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22640a0290 con 0x7f22640a1620 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320531056 0 0) 0x7f22640698e0 con 0x7f2264135560 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient(hunting): my global_id is 4316 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22640a0960 con 0x7f2264135560 2024-05-21T17:21:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3280639662 0 0) 0x7f2250003530 con 0x7f22641359b0 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 >> v1:172.21.15.203:6792/0 conn(0x7f22640a1620 legacy=0x7f226413a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 >> v1:172.21.15.203:6791/0 conn(0x7f2264135560 legacy=0x7f226413dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: _finish_hunting 0 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 monclient: found mon.noname-g 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.203:6790/0 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 --> v1:172.21.15.203:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22640a4bf0 con 0x7f22641359b0 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: _finish_auth 0 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: _check_auth_tickets 2024-05-21T17:21:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2260001840 con 0x7f22641359b0 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: dump: 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 5 monclient: authenticate success, global_id 4233 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 <== mon.3 v1:172.21.15.203:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22600019e0 con 0x7f22641359b0 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 1 -- 172.21.15.3:0/1554278841 <== mon.3 v1:172.21.15.203:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2260003960 con 0x7f22641359b0 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 4 set_mon_vals no callback set 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f2268f4f640 10 monclient: dump: 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient: shutdown 2024-05-21T17:21:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- 172.21.15.3:0/1554278841 >> v1:172.21.15.203:6790/0 conn(0x7f22641359b0 legacy=0x7f2264135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- 172.21.15.3:0/1554278841 shutdown_connections 2024-05-21T17:21:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- 172.21.15.3:0/1554278841 >> 172.21.15.3:0/1554278841 conn(0x7f22640a3ea0 msgr2=0x7f22640a4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- 172.21.15.3:0/1554278841 shutdown_connections 2024-05-21T17:21:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 1 -- 172.21.15.3:0/1554278841 wait complete. 2024-05-21T17:21:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient: build_initial_monmap 2024-05-21T17:21:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.360+0000 7f226c1b5640 10 monclient: monmap: 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 1 Processor -- start 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 1 -- start start 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 10 monclient: init 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 10 monclient: _add_conns ranks=[2,4,0,1,5,8,3,6,7] 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 10 monclient(hunting): picked mon.noname-b con 0x7f2264135560 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 10 monclient(hunting): picked mon.noname-c con 0x7f22641359b0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 10 monclient(hunting): picked mon.noname-a con 0x7f22641d19a0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f226409dc40 con 0x7f22641d19a0 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f226408fe30 con 0x7f2264135560 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22640901e0 con 0x7f22641359b0 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f226c1b5640 10 monclient(hunting): authenticate will time out at 965.524842s 2024-05-21T17:21:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f2269750640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f22641359b0 0x7f226408f720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47506/0 (socket says 172.21.15.3:47506) 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f2269750640 1 -- 172.21.15.3:0/4034422982 learned_addr learned my addr 172.21.15.3:0/4034422982 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1923187632 0 0) 0x7f22640a0960 con 0x7f22641d19a0 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient(hunting): my global_id is 4239 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2248003530 con 0x7f22641d19a0 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 840948915 0 0) 0x7f2248003530 con 0x7f22641d19a0 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22640a0960 con 0x7f22641d19a0 2024-05-21T17:21:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f225c001e20 con 0x7f22641d19a0 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient(hunting): dump: 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:38.387 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2676237966 0 0) 0x7f22640a0960 con 0x7f22641d19a0 2024-05-21T17:21:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 >> v1:172.21.15.3:6791/0 conn(0x7f22641359b0 legacy=0x7f226408f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 >> v1:172.21.15.3:6790/0 conn(0x7f2264135560 legacy=0x7f226408f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 monclient: found mon.a 2024-05-21T17:21:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22640a0480 con 0x7f22641d19a0 2024-05-21T17:21:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.364+0000 7f225a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 5 monclient: authenticate success, global_id 4239 2024-05-21T17:21:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 10 monclient: _renew_subs 2024-05-21T17:21:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2264074890 con 0x7f22641d19a0 2024-05-21T17:21:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 10 monclient: _renew_subs 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2264090d20 con 0x7f22641d19a0 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f225c003c50 con 0x7f22641d19a0 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225a7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f225c004280 con 0x7f22641d19a0 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225bfff640 4 set_mon_vals no callback set 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225a7fc640 10 monclient: dump: 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7f225c004420 con 0x7f22641d19a0 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f224c005090 con 0x7f22641d19a0 2024-05-21T17:21:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (unknown 241544140 0 0) 0x7f225c044ca0 con 0x7f22641d19a0 2024-05-21T17:21:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.368+0000 7f2269f51640 10 monclient: get_auth_request con 0x7f2248038c10 auth_method 0 2024-05-21T17:21:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.372+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f226409dc40 con 0x7f22641d19a0 2024-05-21T17:21:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.372+0000 7f225a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.372+0000 7f225a7fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:38.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.592+0000 7f226c1b5640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-05-21T17:21:38.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.592+0000 7f226c1b5640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-05-21T17:21:38.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.592+0000 7f226c1b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.592+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f224c005380 con 0x7f22641d19a0 2024-05-21T17:21:38.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.596+0000 7f225a7fc640 1 -- 172.21.15.3:0/4034422982 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+86225 (unknown 3797009233 0 1743197671) 0x7f225c014ea0 con 0x7f22641d19a0 2024-05-21T17:21:38.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.596+0000 7f225a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-05-21T17:21:38.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.596+0000 7f225a7fc640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:38.603 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:38.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.600+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 >> v1:172.21.15.203:6812/343827626 conn(0x7f2248038c10 legacy=0x7f224803b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.600+0000 7f226c1b5640 10 monclient: shutdown 2024-05-21T17:21:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.600+0000 7f226c1b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.600+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 >> v1:172.21.15.3:6789/0 conn(0x7f22641d19a0 legacy=0x7f22641d3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.600+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 shutdown_connections 2024-05-21T17:21:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.600+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 >> 172.21.15.3:0/4034422982 conn(0x7f22640a3ea0 msgr2=0x7f22640952d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.600+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 shutdown_connections 2024-05-21T17:21:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.600+0000 7f226c1b5640 1 -- 172.21.15.3:0/4034422982 wait complete. 2024-05-21T17:21:38.621 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":4,"flags":0,"active_gid":4127,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6812","nonce":343827626}]},"active_addr":"172.21.15.203:6812/343827626","active_change":"2024-05-21T17:21:31.046674+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":4106684198}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":4288866933}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":92976687}]}]} 2024-05-21T17:21:38.625 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-21T17:21:38.625 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-21T17:21:38.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-21T17:21:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.740+0000 7f51ec70d640 10 monclient: get_monmap_and_config 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.740+0000 7f51ec70d640 10 monclient: build_initial_monmap 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.740+0000 7f51ec70d640 10 monclient: monmap: 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:38.746988+0000 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:38.746988+0000 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:38.748 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.740+0000 7f51ec70d640 1 Processor -- start 2024-05-21T17:21:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.740+0000 7f51ec70d640 1 -- start start 2024-05-21T17:21:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.740+0000 7f51ec70d640 10 monclient: init 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: _add_conns ranks=[5,2,6,3,1,4,7,0,8] 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): picked mon.noname-h con 0x7f51e40a16d0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): picked mon.noname-b con 0x7f51e4139aa0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): picked mon.noname-d con 0x7f51e413d5b0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e40a21c0 con 0x7f51e4139aa0 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e40a01a0 con 0x7f51e40a16d0 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e4069050 con 0x7f51e413d5b0 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): authenticate will time out at 965.904338s 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51eacaa640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f51e413d5b0 0x7f51e413f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:45904/0 (socket says 172.21.15.3:45904) 2024-05-21T17:21:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51eacaa640 1 -- 172.21.15.3:0/2697344010 learned_addr learned my addr 172.21.15.3:0/2697344010 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1300562551 0 0) 0x7f51e4069050 con 0x7f51e413d5b0 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient(hunting): my global_id is 4191 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51cc003530 con 0x7f51e413d5b0 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2367837268 0 0) 0x7f51e40a01a0 con 0x7f51e40a16d0 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient(hunting): my global_id is 4325 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51e4069050 con 0x7f51e40a16d0 2024-05-21T17:21:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 665795612 0 0) 0x7f51cc003530 con 0x7f51e413d5b0 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 >> v1:172.21.15.203:6791/0 conn(0x7f51e40a16d0 legacy=0x7f51e40a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 >> v1:172.21.15.3:6790/0 conn(0x7f51e4139aa0 legacy=0x7f51e413be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: _finish_hunting 0 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 monclient: found mon.noname-d 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e40a31b0 con 0x7f51e413d5b0 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: _finish_auth 0 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: _check_auth_tickets 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f51dc001840 con 0x7f51e413d5b0 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 5 monclient: authenticate success, global_id 4191 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: dump: 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51dc001a60 con 0x7f51e413d5b0 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 4 set_mon_vals no callback set 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 1 -- 172.21.15.3:0/2697344010 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f51dc004bf0 con 0x7f51e413d5b0 2024-05-21T17:21:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e94a7640 10 monclient: dump: 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: shutdown 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- 172.21.15.3:0/2697344010 >> v1:172.21.15.3:6792/0 conn(0x7f51e413d5b0 legacy=0x7f51e413f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- 172.21.15.3:0/2697344010 shutdown_connections 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- 172.21.15.3:0/2697344010 >> 172.21.15.3:0/2697344010 conn(0x7f51e406c8d0 msgr2=0x7f51e406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- 172.21.15.3:0/2697344010 shutdown_connections 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- 172.21.15.3:0/2697344010 wait complete. 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: build_initial_monmap 2024-05-21T17:21:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: monmap: 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 Processor -- start 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- start start 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: init 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient: _add_conns ranks=[0,3,7,5,8,4,2,6,1] 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): picked mon.noname-a con 0x7f51e40a16d0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): picked mon.noname-g con 0x7f51e4139aa0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): picked mon.noname-i con 0x7f51e413d5b0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e409e220 con 0x7f51e40a16d0 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e4132a60 con 0x7f51e4139aa0 2024-05-21T17:21:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e409dda0 con 0x7f51e413d5b0 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ec70d640 10 monclient(hunting): authenticate will time out at 965.907616s 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ea4a9640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f51e40a16d0 0x7f51e412ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34848/0 (socket says 172.21.15.3:34848) 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51ea4a9640 1 -- 172.21.15.3:0/886509375 learned_addr learned my addr 172.21.15.3:0/886509375 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.744+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3085765959 0 0) 0x7f51e409e220 con 0x7f51e40a16d0 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): my global_id is 4257 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51d0003530 con 0x7f51e40a16d0 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2932077917 0 0) 0x7f51e409dda0 con 0x7f51e413d5b0 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): my global_id is 4255 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51e409e220 con 0x7f51e413d5b0 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2270103593 0 0) 0x7f51d0003530 con 0x7f51e40a16d0 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51e409dda0 con 0x7f51e40a16d0 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f51d4001bc0 con 0x7f51e40a16d0 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2448971892 0 0) 0x7f51e4132a60 con 0x7f51e4139aa0 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient(hunting): my global_id is 4242 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51d0003530 con 0x7f51e4139aa0 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1084663036 0 0) 0x7f51e409dda0 con 0x7f51e40a16d0 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 >> v1:172.21.15.203:6792/0 conn(0x7f51e413d5b0 legacy=0x7f51e41d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 >> v1:172.21.15.203:6790/0 conn(0x7f51e4139aa0 legacy=0x7f51e41d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 monclient: found mon.a 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e4132190 con 0x7f51e40a16d0 2024-05-21T17:21:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 5 monclient: authenticate success, global_id 4257 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 10 monclient: _renew_subs 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f51e4134580 con 0x7f51e40a16d0 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 10 monclient: _renew_subs 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f51e4135520 con 0x7f51e40a16d0 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51d40038d0 con 0x7f51e40a16d0 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f51d4003e80 con 0x7f51e40a16d0 2024-05-21T17:21:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e8ca6640 4 set_mon_vals no callback set 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 10 monclient: dump: 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51c4005090 con 0x7f51e40a16d0 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7f51d40040a0 con 0x7f51e40a16d0 2024-05-21T17:21:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (unknown 241544140 0 0) 0x7f51d4045c90 con 0x7f51e40a16d0 2024-05-21T17:21:38.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.748+0000 7f51e9ca8640 10 monclient: get_auth_request con 0x7f51d0038e50 auth_method 0 2024-05-21T17:21:38.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.752+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f51d4015e90 con 0x7f51e40a16d0 2024-05-21T17:21:38.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.752+0000 7f51e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:38.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.752+0000 7f51e2ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.920+0000 7f51ec70d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.920+0000 7f51ec70d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.920+0000 7f51ec70d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.920+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51c4005380 con 0x7f51e40a16d0 2024-05-21T17:21:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.920+0000 7f51e2ffd640 1 -- 172.21.15.3:0/886509375 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10787 (unknown 92231148 0 4033097912) 0x7f51d4015790 con 0x7f51e40a16d0 2024-05-21T17:21:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.920+0000 7f51e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.920+0000 7f51e2ffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:38.928 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:38.928 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":14,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","created":"2024-05-21T17:21:26.465014+0000","modified":"2024-05-21T17:21:38.196155+0000","last_up_change":"2024-05-21T17:21:35.154389+0000","last_in_change":"2024-05-21T17:21:28.263976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-21T17:21:36.581878+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-21T17:21:37.102413+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7dfaea0b-2159-4f6e-86a6-da6637052771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6800","nonce":857120552}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6801","nonce":857120552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6803","nonce":857120552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6802","nonce":857120552}]},"public_addr":"172.21.15.3:6800/857120552","cluster_addr":"172.21.15.3:6801/857120552","heartbeat_back_addr":"172.21.15.3:6803/857120552","heartbeat_front_addr":"172.21.15.3:6802/857120552","state":["exists","up"]},{"osd":1,"uuid":"28d053ac-034d-43ed-a83e-9ee62a426b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6808","nonce":4137544047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6809","nonce":4137544047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6811","nonce":4137544047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6810","nonce":4137544047}]},"public_addr":"172.21.15.3:6808/4137544047","cluster_addr":"172.21.15.3:6809/4137544047","heartbeat_back_addr":"172.21.15.3:6811/4137544047","heartbeat_front_addr":"172.21.15.3:6810/4137544047","state":["exists","up"]},{"osd":2,"uuid":"9de90871-f66b-4a85-ac48-772541c5b23d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6804","nonce":517111994}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6805","nonce":517111994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6807","nonce":517111994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6806","nonce":517111994}]},"public_addr":"172.21.15.3:6804/517111994","cluster_addr":"172.21.15.3:6805/517111994","heartbeat_back_addr":"172.21.15.3:6807/517111994","heartbeat_front_addr":"172.21.15.3:6806/517111994","state":["exists","up"]},{"osd":3,"uuid":"f8431d10-6318-458b-9a94-51fc47273f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":1468009857}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1468009857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1468009857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":1468009857}]},"public_addr":"172.21.15.203:6804/1468009857","cluster_addr":"172.21.15.203:6805/1468009857","heartbeat_back_addr":"172.21.15.203:6807/1468009857","heartbeat_front_addr":"172.21.15.203:6806/1468009857","state":["exists","up"]},{"osd":4,"uuid":"028748a9-3ca4-4730-960e-8913c9504a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":4020495055}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":4020495055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":4020495055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":4020495055}]},"public_addr":"172.21.15.203:6800/4020495055","cluster_addr":"172.21.15.203:6801/4020495055","heartbeat_back_addr":"172.21.15.203:6803/4020495055","heartbeat_front_addr":"172.21.15.203:6802/4020495055","state":["exists","up"]},{"osd":5,"uuid":"87412b84-97f1-4350-9034-87f8f33da44a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":2732955435}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":2732955435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":2732955435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":2732955435}]},"public_addr":"172.21.15.203:6808/2732955435","cluster_addr":"172.21.15.203:6809/2732955435","heartbeat_back_addr":"172.21.15.203:6811/2732955435","heartbeat_front_addr":"172.21.15.203:6810/2732955435","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.740995+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.776640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.734901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.745181+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.676894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.827672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T17:21:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.924+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 >> v1:172.21.15.203:6812/343827626 conn(0x7f51d0038e50 legacy=0x7f51d003b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.924+0000 7f51ec70d640 10 monclient: shutdown 2024-05-21T17:21:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.924+0000 7f51ec70d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.924+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 >> v1:172.21.15.3:6789/0 conn(0x7f51e40a16d0 legacy=0x7f51e412ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.924+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 shutdown_connections 2024-05-21T17:21:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.924+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 >> 172.21.15.3:0/886509375 conn(0x7f51e406c8d0 msgr2=0x7f51e40a4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.924+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 shutdown_connections 2024-05-21T17:21:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:38.924+0000 7f51ec70d640 1 -- 172.21.15.3:0/886509375 wait complete. 2024-05-21T17:21:38.946 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-21T17:21:38.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.052+0000 7f3499cdc640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.052+0000 7f3499cdc640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.052+0000 7f3499cdc640 10 monclient: monmap: 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.058735+0000 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.058735+0000 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.052+0000 7f3499cdc640 1 Processor -- start 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.052+0000 7f3499cdc640 1 -- start start 2024-05-21T17:21:39.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.052+0000 7f3499cdc640 10 monclient: init 2024-05-21T17:21:39.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: _add_conns ranks=[0,2,3,1,6,8,4,7,5] 2024-05-21T17:21:39.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): picked mon.noname-a con 0x7f3494135bf0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): picked mon.noname-b con 0x7f3494135fd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): picked mon.noname-g con 0x7f349413fc90 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34940a3b80 con 0x7f3494135bf0 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3494068f90 con 0x7f3494135fd0 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34940a4650 con 0x7f349413fc90 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): authenticate will time out at 966.216309s 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3492ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f3494135fd0 0x7f34940a3470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47102/0 (socket says 172.21.15.3:47102) 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3492ffd640 1 -- 172.21.15.3:0/1229772034 learned_addr learned my addr 172.21.15.3:0/1229772034 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434830410 0 0) 0x7f3494068f90 con 0x7f3494135fd0 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): my global_id is 4214 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f347c003530 con 0x7f3494135fd0 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3004382959 0 0) 0x7f34940a3b80 con 0x7f3494135bf0 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): my global_id is 4266 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3494068f90 con 0x7f3494135bf0 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2681399355 0 0) 0x7f34940a4650 con 0x7f349413fc90 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient(hunting): my global_id is 4260 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34940a3b80 con 0x7f349413fc90 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1118361033 0 0) 0x7f347c003530 con 0x7f3494135fd0 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 >> v1:172.21.15.203:6790/0 conn(0x7f349413fc90 legacy=0x7f3494142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 >> v1:172.21.15.3:6789/0 conn(0x7f3494135bf0 legacy=0x7f34940a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 monclient: found mon.noname-b 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349409ce70 con 0x7f3494135fd0 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3488001840 con 0x7f3494135fd0 2024-05-21T17:21:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: dump: 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.070 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 5 monclient: authenticate success, global_id 4214 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f34880019e0 con 0x7f3494135fd0 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 1 -- 172.21.15.3:0/1229772034 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3488004bf0 con 0x7f3494135fd0 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34927fc640 10 monclient: dump: 2024-05-21T17:21:39.071 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.072 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 4 set_mon_vals no callback set 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: shutdown 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- 172.21.15.3:0/1229772034 >> v1:172.21.15.3:6790/0 conn(0x7f3494135fd0 legacy=0x7f34940a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- 172.21.15.3:0/1229772034 shutdown_connections 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- 172.21.15.3:0/1229772034 >> 172.21.15.3:0/1229772034 conn(0x7f349406c8d0 msgr2=0x7f349406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- 172.21.15.3:0/1229772034 shutdown_connections 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- 172.21.15.3:0/1229772034 wait complete. 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: monmap: 2024-05-21T17:21:39.073 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.074 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 Processor -- start 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- start start 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: init 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient: _add_conns ranks=[8,1,0,7,4,2,3,5,6] 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): picked mon.noname-e con 0x7f3494135bf0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): picked mon.noname-f con 0x7f3494135fd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): picked mon.noname-a con 0x7f349413fc90 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34941371f0 con 0x7f349413fc90 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3494136dd0 con 0x7f3494135fd0 2024-05-21T17:21:39.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3494136af0 con 0x7f3494135bf0 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3499cdc640 10 monclient(hunting): authenticate will time out at 966.219673s 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34937fe640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f3494135bf0 0x7f349412a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41632/0 (socket says 172.21.15.3:41632) 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f3493fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f349413fc90 0x7f34941d6280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34858/0 (socket says 172.21.15.3:34858) 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.056+0000 7f34937fe640 1 -- 172.21.15.3:0/3147783069 learned_addr learned my addr 172.21.15.3:0/3147783069 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234784352 0 0) 0x7f34940a3b80 con 0x7f349413fc90 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): my global_id is 4275 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3480003530 con 0x7f349413fc90 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3202009695 0 0) 0x7f3494136af0 con 0x7f3494135bf0 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): my global_id is 4256 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34940a3b80 con 0x7f3494135bf0 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3129259103 0 0) 0x7f3480003530 con 0x7f349413fc90 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3494136af0 con 0x7f349413fc90 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f348c001d80 con 0x7f349413fc90 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2897650994 0 0) 0x7f34940a3b80 con 0x7f3494135bf0 2024-05-21T17:21:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3480003530 con 0x7f3494135bf0 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f34840016a0 con 0x7f3494135bf0 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 670853881 0 0) 0x7f3494136dd0 con 0x7f3494135fd0 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient(hunting): my global_id is 4231 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34940a3b80 con 0x7f3494135fd0 2024-05-21T17:21:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3119333687 0 0) 0x7f3494136af0 con 0x7f349413fc90 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 >> v1:172.21.15.3:6793/0 conn(0x7f3494135bf0 legacy=0x7f349412a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 >> v1:172.21.15.203:6789/0 conn(0x7f3494135fd0 legacy=0x7f349412aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 monclient: found mon.a 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34940a02f0 con 0x7f349413fc90 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 5 monclient: authenticate success, global_id 4275 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 10 monclient: _renew_subs 2024-05-21T17:21:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34940a0580 con 0x7f349413fc90 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 10 monclient: _renew_subs 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f34940a0b90 con 0x7f349413fc90 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f348c002000 con 0x7f349413fc90 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f348c003f90 con 0x7f349413fc90 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3491ffb640 4 set_mon_vals no callback set 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 10 monclient: dump: 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.082 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7f348c004130 con 0x7f349413fc90 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3474005090 con 0x7f349413fc90 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (unknown 241544140 0 0) 0x7f348c045c40 con 0x7f349413fc90 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.060+0000 7f34937fe640 10 monclient: get_auth_request con 0x7f3480039320 auth_method 0 2024-05-21T17:21:39.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.064+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f34941371f0 con 0x7f349413fc90 2024-05-21T17:21:39.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.064+0000 7f347bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.064+0000 7f347bfff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:39.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:39.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:39.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3474005380 con 0x7f349413fc90 2024-05-21T17:21:39.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f347bfff640 1 -- 172.21.15.3:0/3147783069 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10761 (unknown 4221056075 0 2217018945) 0x7f348c015f00 con 0x7f349413fc90 2024-05-21T17:21:39.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f347bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:39.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f347bfff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:39.250 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:39.250 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":15,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","created":"2024-05-21T17:21:26.465014+0000","modified":"2024-05-21T17:21:39.218346+0000","last_up_change":"2024-05-21T17:21:35.154389+0000","last_in_change":"2024-05-21T17:21:28.263976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-21T17:21:36.581878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-21T17:21:37.102413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7dfaea0b-2159-4f6e-86a6-da6637052771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6800","nonce":857120552}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6801","nonce":857120552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6803","nonce":857120552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6802","nonce":857120552}]},"public_addr":"172.21.15.3:6800/857120552","cluster_addr":"172.21.15.3:6801/857120552","heartbeat_back_addr":"172.21.15.3:6803/857120552","heartbeat_front_addr":"172.21.15.3:6802/857120552","state":["exists","up"]},{"osd":1,"uuid":"28d053ac-034d-43ed-a83e-9ee62a426b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6808","nonce":4137544047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6809","nonce":4137544047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6811","nonce":4137544047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6810","nonce":4137544047}]},"public_addr":"172.21.15.3:6808/4137544047","cluster_addr":"172.21.15.3:6809/4137544047","heartbeat_back_addr":"172.21.15.3:6811/4137544047","heartbeat_front_addr":"172.21.15.3:6810/4137544047","state":["exists","up"]},{"osd":2,"uuid":"9de90871-f66b-4a85-ac48-772541c5b23d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6804","nonce":517111994}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6805","nonce":517111994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6807","nonce":517111994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6806","nonce":517111994}]},"public_addr":"172.21.15.3:6804/517111994","cluster_addr":"172.21.15.3:6805/517111994","heartbeat_back_addr":"172.21.15.3:6807/517111994","heartbeat_front_addr":"172.21.15.3:6806/517111994","state":["exists","up"]},{"osd":3,"uuid":"f8431d10-6318-458b-9a94-51fc47273f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":1468009857}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1468009857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1468009857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":1468009857}]},"public_addr":"172.21.15.203:6804/1468009857","cluster_addr":"172.21.15.203:6805/1468009857","heartbeat_back_addr":"172.21.15.203:6807/1468009857","heartbeat_front_addr":"172.21.15.203:6806/1468009857","state":["exists","up"]},{"osd":4,"uuid":"028748a9-3ca4-4730-960e-8913c9504a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":4020495055}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":4020495055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":4020495055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":4020495055}]},"public_addr":"172.21.15.203:6800/4020495055","cluster_addr":"172.21.15.203:6801/4020495055","heartbeat_back_addr":"172.21.15.203:6803/4020495055","heartbeat_front_addr":"172.21.15.203:6802/4020495055","state":["exists","up"]},{"osd":5,"uuid":"87412b84-97f1-4350-9034-87f8f33da44a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":2732955435}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":2732955435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":2732955435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":2732955435}]},"public_addr":"172.21.15.203:6808/2732955435","cluster_addr":"172.21.15.203:6809/2732955435","heartbeat_back_addr":"172.21.15.203:6811/2732955435","heartbeat_front_addr":"172.21.15.203:6810/2732955435","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.740995+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.776640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.734901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.745181+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.676894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.827672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T17:21:39.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 >> v1:172.21.15.203:6812/343827626 conn(0x7f3480039320 legacy=0x7f348003b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 10 monclient: shutdown 2024-05-21T17:21:39.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 >> v1:172.21.15.3:6789/0 conn(0x7f349413fc90 legacy=0x7f34941d6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 shutdown_connections 2024-05-21T17:21:39.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 >> 172.21.15.3:0/3147783069 conn(0x7f349406c8d0 msgr2=0x7f3494130e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 shutdown_connections 2024-05-21T17:21:39.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.244+0000 7f3499cdc640 1 -- 172.21.15.3:0/3147783069 wait complete. 2024-05-21T17:21:39.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-21T17:21:39.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-21T17:21:39.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-21T17:21:39.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-21T17:21:39.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-21T17:21:39.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-21T17:21:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient: monmap: 2024-05-21T17:21:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.389503+0000 2024-05-21T17:21:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.389503+0000 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.390 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 1 Processor -- start 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient: monmap: 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.389934+0000 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.389934+0000 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.393 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 1 -- start start 2024-05-21T17:21:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient: init 2024-05-21T17:21:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 1 Processor -- start 2024-05-21T17:21:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient: _add_conns ranks=[2,0,6,5,4,3,8,1,7] 2024-05-21T17:21:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient(hunting): picked mon.noname-b con 0x7feb980a5bd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient(hunting): picked mon.noname-a con 0x7feb980a5fb0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient(hunting): picked mon.noname-d con 0x7feb981431a0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb980a7980 con 0x7feb980a5fb0 2024-05-21T17:21:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb9809cce0 con 0x7feb980a5bd0 2024-05-21T17:21:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb9809cfc0 con 0x7feb981431a0 2024-05-21T17:21:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb9d914640 10 monclient(hunting): authenticate will time out at 966.546915s 2024-05-21T17:21:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb967fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7feb980a5fb0 0x7feb980a6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34872/0 (socket says 172.21.15.3:34872) 2024-05-21T17:21:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 1 -- start start 2024-05-21T17:21:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient: init 2024-05-21T17:21:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb96ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7feb980a5bd0 0x7feb9809d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47116/0 (socket says 172.21.15.3:47116) 2024-05-21T17:21:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb96ffd640 1 -- 172.21.15.3:0/2757061146 learned_addr learned my addr 172.21.15.3:0/2757061146 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient: _add_conns ranks=[4,1,2,7,8,6,5,3,0] 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient(hunting): picked mon.noname-c con 0x7f4be80a5bd0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient(hunting): picked mon.noname-f con 0x7f4be80a66c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient(hunting): picked mon.noname-b con 0x7f4be80a70c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be809e480 con 0x7f4be80a66c0 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be809cce0 con 0x7f4be80a70c0 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be809cfc0 con 0x7f4be80a5bd0 2024-05-21T17:21:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7f4bedf99640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2010570567 0 0) 0x7feb9809cfc0 con 0x7feb981431a0 2024-05-21T17:21:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb95ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb95ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.384+0000 7feb95ffb640 10 monclient(hunting): my global_id is 4209 2024-05-21T17:21:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb80003530 con 0x7feb981431a0 2024-05-21T17:21:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 10 monclient(hunting): authenticate will time out at 966.548031s 2024-05-21T17:21:39.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be7fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f4be80a70c0 0x7f4be8141370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47124/0 (socket says 172.21.15.3:47124) 2024-05-21T17:21:39.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be7fff640 1 -- 172.21.15.3:0/1260617604 learned_addr learned my addr 172.21.15.3:0/1260617604 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1558009510 0 0) 0x7feb9809cce0 con 0x7feb980a5bd0 2024-05-21T17:21:39.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient(hunting): my global_id is 4223 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb9809cfc0 con 0x7feb980a5bd0 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3735209128 0 0) 0x7feb980a7980 con 0x7feb980a5fb0 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient(hunting): my global_id is 4284 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb9809cce0 con 0x7feb980a5fb0 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2880542849 0 0) 0x7feb80003530 con 0x7feb981431a0 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 >> v1:172.21.15.3:6790/0 conn(0x7feb980a5bd0 legacy=0x7feb9809d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 >> v1:172.21.15.3:6789/0 conn(0x7feb980a5fb0 legacy=0x7feb980a6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 monclient: found mon.noname-d 2024-05-21T17:21:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9809d2a0 con 0x7feb981431a0 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feb90001bc0 con 0x7feb981431a0 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: dump: 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.401 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3242148445 0 0) 0x7f4be809cce0 con 0x7f4be80a70c0 2024-05-21T17:21:39.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): my global_id is 4232 2024-05-21T17:21:39.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bd0003530 con 0x7f4be80a70c0 2024-05-21T17:21:39.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb90001d60 con 0x7feb981431a0 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 1 -- 172.21.15.3:0/2757061146 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feb90001f90 con 0x7feb981431a0 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb95ffb640 10 monclient: dump: 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.403 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.404 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.404 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.404 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.404 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.404 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.404 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.404 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465763950 0 0) 0x7f4be809e480 con 0x7f4be80a66c0 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): my global_id is 4240 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4be809cce0 con 0x7f4be80a66c0 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2361083239 0 0) 0x7f4be809cfc0 con 0x7f4be80a5bd0 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient(hunting): my global_id is 4279 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4be809e480 con 0x7f4be80a5bd0 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3745427090 0 0) 0x7f4bd0003530 con 0x7f4be80a70c0 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 >> v1:172.21.15.3:6791/0 conn(0x7f4be80a5bd0 legacy=0x7f4be80a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 >> v1:172.21.15.203:6789/0 conn(0x7f4be80a66c0 legacy=0x7f4be813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 monclient: found mon.noname-b 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be80742e0 con 0x7f4be80a70c0 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 5 monclient: authenticate success, global_id 4232 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4be0001bc0 con 0x7f4be80a70c0 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient: dump: 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 5 monclient: authenticate success, global_id 4209 2024-05-21T17:21:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 4 set_mon_vals no callback set 2024-05-21T17:21:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 10 monclient: shutdown 2024-05-21T17:21:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 1 -- 172.21.15.3:0/2757061146 >> v1:172.21.15.3:6792/0 conn(0x7feb981431a0 legacy=0x7feb98145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 1 -- 172.21.15.3:0/2757061146 shutdown_connections 2024-05-21T17:21:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 1 -- 172.21.15.3:0/2757061146 >> 172.21.15.3:0/2757061146 conn(0x7feb980ab7d0 msgr2=0x7feb980a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 1 -- 172.21.15.3:0/2757061146 shutdown_connections 2024-05-21T17:21:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 1 -- 172.21.15.3:0/1260617604 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4be0001d60 con 0x7f4be80a70c0 2024-05-21T17:21:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4be67fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 1 -- 172.21.15.3:0/2757061146 wait complete. 2024-05-21T17:21:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7feb9d914640 10 monclient: monmap: 2024-05-21T17:21:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.411 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 4 set_mon_vals no callback set 2024-05-21T17:21:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 10 monclient: shutdown 2024-05-21T17:21:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.388+0000 7f4bedf99640 1 -- 172.21.15.3:0/1260617604 >> v1:172.21.15.3:6790/0 conn(0x7f4be80a70c0 legacy=0x7f4be8141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 -- 172.21.15.3:0/1260617604 shutdown_connections 2024-05-21T17:21:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 -- 172.21.15.3:0/1260617604 >> 172.21.15.3:0/1260617604 conn(0x7f4be80ab7d0 msgr2=0x7f4be80a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 1 Processor -- start 2024-05-21T17:21:39.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 -- 172.21.15.3:0/1260617604 shutdown_connections 2024-05-21T17:21:39.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 1 -- start start 2024-05-21T17:21:39.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient: init 2024-05-21T17:21:39.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient: _add_conns ranks=[7,2,0,6,5,1,8,4,3] 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient(hunting): picked mon.noname-i con 0x7feb980a5bd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient(hunting): picked mon.noname-b con 0x7feb980a5fb0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient(hunting): picked mon.noname-a con 0x7feb981431a0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb9806f4e0 con 0x7feb981431a0 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb98139f40 con 0x7feb980a5fb0 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb9809e380 con 0x7feb980a5bd0 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient(hunting): authenticate will time out at 966.553623s 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 -- 172.21.15.3:0/1260617604 wait complete. 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient: monmap: 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.415 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.416 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.416 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.416 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.416 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.416 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 Processor -- start 2024-05-21T17:21:39.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb967fc640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7feb980a5fb0 0x7feb98139830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47136/0 (socket says 172.21.15.3:47136) 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb967fc640 1 -- 172.21.15.3:0/3199525080 learned_addr learned my addr 172.21.15.3:0/3199525080 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076193059 0 0) 0x7feb98139f40 con 0x7feb980a5fb0 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): my global_id is 4241 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb80003530 con 0x7feb980a5fb0 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2582539209 0 0) 0x7feb9806f4e0 con 0x7feb981431a0 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): my global_id is 4293 2024-05-21T17:21:39.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb98139f40 con 0x7feb981431a0 2024-05-21T17:21:39.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 -- start start 2024-05-21T17:21:39.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient: init 2024-05-21T17:21:39.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient: _add_conns ranks=[0,5,6,2,4,1,3,8,7] 2024-05-21T17:21:39.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient(hunting): picked mon.noname-a con 0x7f4be80a5bd0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient(hunting): picked mon.noname-h con 0x7f4be80a66c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient(hunting): picked mon.noname-d con 0x7f4be80a70c0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be8071ad0 con 0x7f4be80a5bd0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be813e3a0 con 0x7f4be80a66c0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1997014136 0 0) 0x7feb80003530 con 0x7feb980a5fb0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb9806f4e0 con 0x7feb980a5fb0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feb8c001ba0 con 0x7feb980a5fb0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.420 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be8072610 con 0x7f4be80a70c0 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4bedf99640 10 monclient(hunting): authenticate will time out at 966.555040s 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4be7fff640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f4be80a70c0 0x7f4be81da4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:45926/0 (socket says 172.21.15.3:45926) 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4be7fff640 1 -- 172.21.15.3:0/978797953 learned_addr learned my addr 172.21.15.3:0/978797953 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7f4be77fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4be80a5bd0 0x7f4be8138a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34902/0 (socket says 172.21.15.3:34902) 2024-05-21T17:21:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 274376166 0 0) 0x7feb98139f40 con 0x7feb981431a0 2024-05-21T17:21:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb80003530 con 0x7feb981431a0 2024-05-21T17:21:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feb90001a80 con 0x7feb981431a0 2024-05-21T17:21:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3420228366 0 0) 0x7feb9806f4e0 con 0x7feb980a5fb0 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 >> v1:172.21.15.203:6792/0 conn(0x7feb980a5bd0 legacy=0x7feb98139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 >> v1:172.21.15.3:6789/0 conn(0x7feb981431a0 legacy=0x7feb981da130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 monclient: found mon.b 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9806f200 con 0x7feb980a5fb0 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 5 monclient: authenticate success, global_id 4241 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient: _renew_subs 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb980a4af0 con 0x7feb980a5fb0 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient: _renew_subs 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feb980a5470 con 0x7feb980a5fb0 2024-05-21T17:21:39.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.392+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb8c004aa0 con 0x7feb980a5fb0 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feb8c004f90 con 0x7feb980a5fb0 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 10 monclient: dump: 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb957fa640 4 set_mon_vals no callback set 2024-05-21T17:21:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611542186 0 0) 0x7f4be8072610 con 0x7f4be80a70c0 2024-05-21T17:21:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): my global_id is 4218 2024-05-21T17:21:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bd0003530 con 0x7f4be80a70c0 2024-05-21T17:21:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb9d914640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7feb78000f90 con 0x7feb980a5fb0 2024-05-21T17:21:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774621278 0 0) 0x7f4be8071ad0 con 0x7f4be80a5bd0 2024-05-21T17:21:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): my global_id is 4302 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4be8072610 con 0x7f4be80a5bd0 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205016175 0 0) 0x7f4be809cce0 con 0x7f4be80a66c0 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): my global_id is 4343 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4be8071ad0 con 0x7f4be80a66c0 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3236101082 0 0) 0x7f4bd0003530 con 0x7f4be80a70c0 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4be809cce0 con 0x7f4be80a70c0 2024-05-21T17:21:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4be0001fd0 con 0x7f4be80a70c0 2024-05-21T17:21:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7feb8c0100e0 con 0x7feb980a5fb0 2024-05-21T17:21:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.434 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.434 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.434 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 367194545 0 0) 0x7f4be8071ad0 con 0x7f4be80a66c0 2024-05-21T17:21:39.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7feb8c045b10 con 0x7feb980a5fb0 2024-05-21T17:21:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4bd0003530 con 0x7f4be80a66c0 2024-05-21T17:21:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 718690999 0 0) 0x7f4be809cce0 con 0x7f4be80a70c0 2024-05-21T17:21:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 >> v1:172.21.15.203:6791/0 conn(0x7f4be80a66c0 legacy=0x7f4be81d6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6804/517111994 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb80043ce0 con 0x7feb80040070 2024-05-21T17:21:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (unknown 2942983318 0 0) 0x7feb8c045e90 con 0x7feb980a5fb0 2024-05-21T17:21:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb7ffff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-05-21T17:21:39.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 >> v1:172.21.15.3:6789/0 conn(0x7f4be80a5bd0 legacy=0x7f4be8138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 monclient: found mon.d 2024-05-21T17:21:39.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be809e6a0 con 0x7f4be80a70c0 2024-05-21T17:21:39.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb96ffd640 10 monclient: get_auth_request con 0x7feb80039470 auth_method 0 2024-05-21T17:21:39.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7feb977fe640 10 monclient: get_auth_request con 0x7feb80040070 auth_method 0 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bedf99640 5 monclient: authenticate success, global_id 4218 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bedf99640 10 monclient: _renew_subs 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bedf99640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4be80a79f0 con 0x7f4be80a70c0 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bedf99640 10 monclient: _renew_subs 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bedf99640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.396+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f4be8072330 con 0x7f4be80a70c0 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bedf99640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4be806bf50 con 0x7f4be80a70c0 2024-05-21T17:21:39.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4be0001b60 con 0x7f4be80a70c0 2024-05-21T17:21:39.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== osd.2 v1:172.21.15.3:6804/517111994 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7feb80043ce0 con 0x7feb80040070 2024-05-21T17:21:39.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4be5ffb640 4 set_mon_vals no callback set 2024-05-21T17:21:39.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4be0004f70 con 0x7f4be80a70c0 2024-05-21T17:21:39.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 10 monclient: dump: 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.441 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7f4be0010730 con 0x7f4be80a70c0 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f4be0045cc0 con 0x7f4be80a70c0 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6808/4137544047 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4be0001b60 con 0x7f4bd0040200 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== mon.6 v1:172.21.15.3:6792/0 9 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (unknown 2942983318 0 0) 0x7f4be0046020 con 0x7f4be80a70c0 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4be77fe640 10 monclient: get_auth_request con 0x7f4bd0039550 auth_method 0 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4be6ffd640 10 monclient: get_auth_request con 0x7f4bd0040200 auth_method 0 2024-05-21T17:21:39.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.400+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== osd.1 v1:172.21.15.3:6808/4137544047 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4be0001b60 con 0x7f4bd0040200 2024-05-21T17:21:39.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.416+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 --> v1:172.21.15.3:6808/4137544047 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4be8077d50 con 0x7f4bd0040200 2024-05-21T17:21:39.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.416+0000 7f4bcffff640 1 -- 172.21.15.3:0/978797953 <== osd.1 v1:172.21.15.3:6808/4137544047 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1800444376) 0x7f4be8077d50 con 0x7f4bd0040200 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.416+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 >> v1:172.21.15.3:6808/4137544047 conn(0x7f4bd0040200 legacy=0x7f4bd0042620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.416+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 >> v1:172.21.15.203:6812/343827626 conn(0x7f4bd0039550 legacy=0x7f4bd003ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.416+0000 7f4bedf99640 10 monclient: shutdown 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.416+0000 7f4bedf99640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.416+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 >> v1:172.21.15.3:6792/0 conn(0x7f4be80a70c0 legacy=0x7f4be81da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.416+0000 7f4be7fff640 1 -- 172.21.15.3:0/978797953 reap_dead start 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.420+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 shutdown_connections 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.420+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 >> 172.21.15.3:0/978797953 conn(0x7f4be80ab7d0 msgr2=0x7f4be813bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.420+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 shutdown_connections 2024-05-21T17:21:39.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.420+0000 7f4bedf99640 1 -- 172.21.15.3:0/978797953 wait complete. 2024-05-21T17:21:39.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 --> v1:172.21.15.3:6804/517111994 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feb78002c00 con 0x7feb80040070 2024-05-21T17:21:39.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb7ffff640 1 -- 172.21.15.3:0/3199525080 <== osd.2 v1:172.21.15.3:6804/517111994 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1800444376) 0x7feb78002c00 con 0x7feb80040070 2024-05-21T17:21:39.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 >> v1:172.21.15.3:6804/517111994 conn(0x7feb80040070 legacy=0x7feb80042490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 >> v1:172.21.15.203:6812/343827626 conn(0x7feb80039470 legacy=0x7feb8003b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 10 monclient: shutdown 2024-05-21T17:21:39.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 >> v1:172.21.15.3:6790/0 conn(0x7feb980a5fb0 legacy=0x7feb98139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb977fe640 1 -- 172.21.15.3:0/3199525080 reap_dead start 2024-05-21T17:21:39.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 shutdown_connections 2024-05-21T17:21:39.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 >> 172.21.15.3:0/3199525080 conn(0x7feb980ab7d0 msgr2=0x7feb9813c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 shutdown_connections 2024-05-21T17:21:39.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7feb9d914640 1 -- 172.21.15.3:0/3199525080 wait complete. 2024-05-21T17:21:39.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7fab50fc9640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7fab50fc9640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7fab50fc9640 10 monclient: monmap: 2024-05-21T17:21:39.453 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.453 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.435141+0000 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.435141+0000 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.454 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.428+0000 7fab50fc9640 1 Processor -- start 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 1 -- start start 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient: init 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient: _add_conns ranks=[1,5,7,3,0,4,6,2,8] 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient(hunting): picked mon.noname-f con 0x7fab4c0a0ac0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient(hunting): picked mon.noname-h con 0x7fab4c0a0ea0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient(hunting): picked mon.noname-i con 0x7fab4c0a24e0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4c09e1c0 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4c09e670 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4c0a7960 con 0x7fab4c0a24e0 2024-05-21T17:21:39.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient(hunting): authenticate will time out at 966.592676s 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab4a59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fab4c0a0ac0 0x7fab4c0a16c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.3:54746/0 (socket says 172.21.15.3:54746) 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab4a59c640 1 -- 172.21.15.3:0/3757965962 learned_addr learned my addr 172.21.15.3:0/3757965962 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738367583 0 0) 0x7fab4c09e1c0 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient(hunting): my global_id is 4249 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab38003510 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 921651633 0 0) 0x7fab4c09e670 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient(hunting): my global_id is 4352 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab4c09e1c0 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2112521122 0 0) 0x7fab38003510 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 >> v1:172.21.15.203:6792/0 conn(0x7fab4c0a24e0 legacy=0x7fab4c14e3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 >> v1:172.21.15.203:6791/0 conn(0x7fab4c0a0ea0 legacy=0x7fab4c0a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 monclient: found mon.noname-f 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.203:6789/0 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4c0a2d60 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 5 monclient: authenticate success, global_id 4249 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fab3c001460 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: dump: 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.458 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab3c001670 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 1 -- 172.21.15.3:0/3757965962 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fab3c001be0 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab49d9b640 10 monclient: dump: 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.460 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 4 set_mon_vals no callback set 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 10 monclient: shutdown 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.432+0000 7fab50fc9640 1 -- 172.21.15.3:0/3757965962 >> v1:172.21.15.203:6789/0 conn(0x7fab4c0a0ac0 legacy=0x7fab4c0a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 -- 172.21.15.3:0/3757965962 shutdown_connections 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 -- 172.21.15.3:0/3757965962 >> 172.21.15.3:0/3757965962 conn(0x7fab4c0a72c0 msgr2=0x7fab4c0a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 -- 172.21.15.3:0/3757965962 shutdown_connections 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 -- 172.21.15.3:0/3757965962 wait complete. 2024-05-21T17:21:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient: monmap: 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 Processor -- start 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 -- start start 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient: init 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient: _add_conns ranks=[7,8,5,3,6,4,0,1,2] 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient(hunting): picked mon.noname-i con 0x7fab4c0a0ac0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient(hunting): picked mon.noname-e con 0x7fab4c0a0ea0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient(hunting): picked mon.noname-h con 0x7fab4c0a24e0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4c070a10 con 0x7fab4c0a24e0 2024-05-21T17:21:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4c073b30 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4c0a30c0 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab50fc9640 10 monclient(hunting): authenticate will time out at 966.598681s 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab41d9b640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fab4c0a0ea0 0x7fab4c141380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41640/0 (socket says 172.21.15.3:41640) 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab41d9b640 1 -- 172.21.15.3:0/1009608127 learned_addr learned my addr 172.21.15.3:0/1009608127 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3813698497 0 0) 0x7fab4c0a30c0 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): my global_id is 4265 2024-05-21T17:21:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab2c003530 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.465 INFO:teuthology.orchestra.run.smithi003.stdout:47244640259 2024-05-21T17:21:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 876673265 0 0) 0x7fab4c073b30 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): my global_id is 4282 2024-05-21T17:21:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab4c0a30c0 con 0x7fab4c0a0ac0 2024-05-21T17:21:39.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3154604199 0 0) 0x7fab2c003530 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-21T17:21:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab4c073b30 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fab34001bc0 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): dump: 2024-05-21T17:21:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.469 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.436+0000 7fab437fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2751210621 0 0) 0x7fab4c073b30 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 >> v1:172.21.15.203:6792/0 conn(0x7fab4c0a0ac0 legacy=0x7fab4c140c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 >> v1:172.21.15.203:6791/0 conn(0x7fab4c0a24e0 legacy=0x7fab4c141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 monclient: found mon.e 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4c142850 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 5 monclient: authenticate success, global_id 4265 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 10 monclient: _renew_subs 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fab4c070310 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 10 monclient: _renew_subs 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fab4c0735a0 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab340038b0 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.3:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fab4c06bf50 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fab34003de0 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: dump: 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab4959a640 4 set_mon_vals no callback set 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7fab34004040 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7fab34045b60 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.203:6800/4020495055 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab2c0435b0 con 0x7fab2c03f940 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== mon.8 v1:172.21.15.3:6793/0 9 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (unknown 2942983318 0 0) 0x7fab34045f20 con 0x7fab4c0a0ea0 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab4a59c640 10 monclient: get_auth_request con 0x7fab2c038d40 auth_method 0 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab437fe640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.440+0000 7fab4ad9d640 10 monclient: get_auth_request con 0x7fab2c03f940 auth_method 0 2024-05-21T17:21:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.444+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== osd.4 v1:172.21.15.203:6800/4020495055 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fab2c0435b0 con 0x7fab2c03f940 2024-05-21T17:21:39.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.448+0000 7f69d7da4640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.448+0000 7f69d7da4640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.448+0000 7f69d7da4640 10 monclient: monmap: 2024-05-21T17:21:39.476 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.455709+0000 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.455709+0000 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 Processor -- start 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- start start 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient: init 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient: _add_conns ranks=[6,7,8,5,3,1,2,4,0] 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient(hunting): picked mon.noname-d con 0x7f69d00a5bd0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient(hunting): picked mon.noname-i con 0x7f69d00a66c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient(hunting): picked mon.noname-e con 0x7f69d00a70c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d009e480 con 0x7f69d00a5bd0 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d009cce0 con 0x7f69d00a66c0 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d009cfc0 con 0x7f69d00a70c0 2024-05-21T17:21:39.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient(hunting): authenticate will time out at 966.612945s 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d5b40640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f69d00a5bd0 0x7f69d00a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:45934/0 (socket says 172.21.15.3:45934) 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d5b40640 1 -- 172.21.15.3:0/1512240136 learned_addr learned my addr 172.21.15.3:0/1512240136 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660198131 0 0) 0x7f69d009cfc0 con 0x7f69d00a70c0 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): my global_id is 4274 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69bc003530 con 0x7f69d00a70c0 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1145060263 0 0) 0x7f69d009e480 con 0x7f69d00a5bd0 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): my global_id is 4227 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69d009cfc0 con 0x7f69d00a5bd0 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864935266 0 0) 0x7f69d009cce0 con 0x7f69d00a66c0 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient(hunting): my global_id is 4291 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69d009e480 con 0x7f69d00a66c0 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2339819809 0 0) 0x7f69d009cfc0 con 0x7f69d00a5bd0 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 >> v1:172.21.15.3:6793/0 conn(0x7f69d00a70c0 legacy=0x7f69d0141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 >> v1:172.21.15.203:6792/0 conn(0x7f69d00a66c0 legacy=0x7f69d013dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 monclient: found mon.noname-d 2024-05-21T17:21:39.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d00742e0 con 0x7f69d00a5bd0 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 5 monclient: authenticate success, global_id 4227 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f69cc001820 con 0x7f69d00a5bd0 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: dump: 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.481 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.482 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69cc001a80 con 0x7f69d00a5bd0 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 1 -- 172.21.15.3:0/1512240136 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f69cc004bf0 con 0x7f69d00a5bd0 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 4 set_mon_vals no callback set 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d4b3e640 10 monclient: dump: 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.483 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient: shutdown 2024-05-21T17:21:39.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- 172.21.15.3:0/1512240136 >> v1:172.21.15.3:6792/0 conn(0x7f69d00a5bd0 legacy=0x7f69d00a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- 172.21.15.3:0/1512240136 shutdown_connections 2024-05-21T17:21:39.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- 172.21.15.3:0/1512240136 >> 172.21.15.3:0/1512240136 conn(0x7f69d00ab7d0 msgr2=0x7f69d00a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.485 INFO:teuthology.orchestra.run.smithi003.stdout:47244640259 2024-05-21T17:21:39.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-21T17:21:39.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- 172.21.15.3:0/1512240136 shutdown_connections 2024-05-21T17:21:39.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 1 -- 172.21.15.3:0/1512240136 wait complete. 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.452+0000 7f69d7da4640 10 monclient: monmap: 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.487 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 1 Processor -- start 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient: monmap: 2024-05-21T17:21:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.460331+0000 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.460331+0000 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.489 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.490 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 1 -- start start 2024-05-21T17:21:39.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient: init 2024-05-21T17:21:39.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient: _add_conns ranks=[5,1,2,7,4,3,8,6,0] 2024-05-21T17:21:39.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient(hunting): picked mon.noname-h con 0x7f69d00a5bd0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient(hunting): picked mon.noname-f con 0x7f69d00a66c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 1 Processor -- start 2024-05-21T17:21:39.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient(hunting): picked mon.noname-b con 0x7f69d00a70c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d0141980 con 0x7f69d00a66c0 2024-05-21T17:21:39.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d01218b0 con 0x7f69d00a70c0 2024-05-21T17:21:39.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d01221d0 con 0x7f69d00a5bd0 2024-05-21T17:21:39.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient(hunting): authenticate will time out at 966.616879s 2024-05-21T17:21:39.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d6341640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f69d00a70c0 0x7f69d01d5d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47148/0 (socket says 172.21.15.3:47148) 2024-05-21T17:21:39.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d6341640 1 -- 172.21.15.3:0/1952714645 learned_addr learned my addr 172.21.15.3:0/1952714645 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 1 -- start start 2024-05-21T17:21:39.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient: init 2024-05-21T17:21:39.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2986440500 0 0) 0x7f69d01218b0 con 0x7f69d00a70c0 2024-05-21T17:21:39.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): my global_id is 4250 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b4003530 con 0x7f69d00a70c0 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient: monmap: 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.461477+0000 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.461477+0000 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.497 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.498 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.498 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.498 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.498 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.498 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.498 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3153783481 0 0) 0x7f69b4003530 con 0x7f69d00a70c0 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69d01218b0 con 0x7f69d00a70c0 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639869947 0 0) 0x7f69d0141980 con 0x7f69d00a66c0 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): my global_id is 4258 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b4003530 con 0x7f69d00a66c0 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f69c8001de0 con 0x7f69d00a70c0 2024-05-21T17:21:39.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient: _add_conns ranks=[0,3,7,8,4,1,6,5,2] 2024-05-21T17:21:39.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): dump: 2024-05-21T17:21:39.500 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.500 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.500 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.501 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient(hunting): picked mon.noname-a con 0x7f91400a5bd0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 692479554 0 0) 0x7f69d01221d0 con 0x7f69d00a5bd0 2024-05-21T17:21:39.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient(hunting): picked mon.noname-g con 0x7f91400a66c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:39.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient(hunting): my global_id is 4370 2024-05-21T17:21:39.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69d0141980 con 0x7f69d00a5bd0 2024-05-21T17:21:39.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient(hunting): picked mon.noname-i con 0x7f91400a70c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914009e480 con 0x7f91400a5bd0 2024-05-21T17:21:39.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914009cce0 con 0x7f91400a66c0 2024-05-21T17:21:39.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914009cfc0 con 0x7f91400a70c0 2024-05-21T17:21:39.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient(hunting): authenticate will time out at 966.617910s 2024-05-21T17:21:39.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 1 Processor -- start 2024-05-21T17:21:39.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913effd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f91400a5bd0 0x7f91400a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34904/0 (socket says 172.21.15.3:34904) 2024-05-21T17:21:39.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913effd640 1 -- 172.21.15.3:0/3245095981 learned_addr learned my addr 172.21.15.3:0/3245095981 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3764364785 0 0) 0x7f69d01218b0 con 0x7f69d00a70c0 2024-05-21T17:21:39.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 >> v1:172.21.15.203:6791/0 conn(0x7f69d00a5bd0 legacy=0x7f69d013b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 >> v1:172.21.15.203:6789/0 conn(0x7f69d00a66c0 legacy=0x7f69d01d2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 monclient: found mon.b 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d01224b0 con 0x7f69d00a70c0 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 5 monclient: authenticate success, global_id 4250 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient: _renew_subs 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69d0122790 con 0x7f69d00a70c0 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient: _renew_subs 2024-05-21T17:21:39.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f69d01424e0 con 0x7f69d00a70c0 2024-05-21T17:21:39.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 1 -- start start 2024-05-21T17:21:39.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient: init 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3813230677 0 0) 0x7f914009e480 con 0x7f91400a5bd0 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient(hunting): my global_id is 4311 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9128003530 con 0x7f91400a5bd0 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2151714119 0 0) 0x7f9128003530 con 0x7f91400a5bd0 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 >> v1:172.21.15.203:6792/0 conn(0x7f91400a70c0 legacy=0x7f9140141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 >> v1:172.21.15.203:6790/0 conn(0x7f91400a66c0 legacy=0x7f914013dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 monclient: found mon.noname-a 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91400742e0 con 0x7f91400a5bd0 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69c8003c30 con 0x7f69d00a70c0 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f69c8001f80 con 0x7f69d00a70c0 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c67fc640 10 monclient: dump: 2024-05-21T17:21:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9130001840 con 0x7f91400a5bd0 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: dump: 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.512 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient: _add_conns ranks=[8,7,0,5,4,2,1,6,3] 2024-05-21T17:21:39.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient(hunting): picked mon.noname-e con 0x7f1814135bf0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:39.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient(hunting): picked mon.noname-i con 0x7f1814135fd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f91300019e0 con 0x7f91400a5bd0 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 1 -- 172.21.15.3:0/3245095981 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9130001cd0 con 0x7f91400a5bd0 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f913dffb640 10 monclient: dump: 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.514 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 5 monclient: authenticate success, global_id 4311 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f91457eb640 4 set_mon_vals no callback set 2024-05-21T17:21:39.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient(hunting): picked mon.noname-a con 0x7f181413fc90 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18140a3b80 con 0x7f181413fc90 2024-05-21T17:21:39.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1814068f90 con 0x7f1814135fd0 2024-05-21T17:21:39.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18140a4650 con 0x7f1814135bf0 2024-05-21T17:21:39.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1819029640 10 monclient(hunting): authenticate will time out at 966.619223s 2024-05-21T17:21:39.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1812d9d640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f1814135bf0 0x7f18140a2d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41670/0 (socket says 172.21.15.3:41670) 2024-05-21T17:21:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1812d9d640 1 -- 172.21.15.3:0/1289398666 learned_addr learned my addr 172.21.15.3:0/1289398666 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1421515485 0 0) 0x7f18140a4650 con 0x7f1814135bf0 2024-05-21T17:21:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient(hunting): my global_id is 4283 2024-05-21T17:21:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17fc003530 con 0x7f1814135bf0 2024-05-21T17:21:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f69c7fff640 4 set_mon_vals no callback set 2024-05-21T17:21:39.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3097340869 0 0) 0x7f17fc003530 con 0x7f1814135bf0 2024-05-21T17:21:39.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 >> v1:172.21.15.203:6792/0 conn(0x7f1814135fd0 legacy=0x7f18140a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 >> v1:172.21.15.3:6789/0 conn(0x7f181413fc90 legacy=0x7f1814142080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:21:39.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 1 monclient: found mon.noname-e 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181409ce70 con 0x7f1814135bf0 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1804001840 con 0x7f1814135bf0 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.456+0000 7f1811d9b640 10 monclient: dump: 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.520 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18040019e0 con 0x7f1814135bf0 2024-05-21T17:21:39.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1811d9b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1811d9b640 1 -- 172.21.15.3:0/1289398666 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1804004bf0 con 0x7f1814135bf0 2024-05-21T17:21:39.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1811d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1811d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:39.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: shutdown 2024-05-21T17:21:39.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- 172.21.15.3:0/3245095981 >> v1:172.21.15.3:6789/0 conn(0x7f91400a5bd0 legacy=0x7f91400a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1811d9b640 10 monclient: dump: 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.522 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 5 monclient: authenticate success, global_id 4283 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 4 set_mon_vals no callback set 2024-05-21T17:21:39.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- 172.21.15.3:0/3245095981 shutdown_connections 2024-05-21T17:21:39.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- 172.21.15.3:0/3245095981 >> 172.21.15.3:0/3245095981 conn(0x7f91400ab7d0 msgr2=0x7f91400a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- 172.21.15.3:0/3245095981 shutdown_connections 2024-05-21T17:21:39.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- 172.21.15.3:0/3245095981 wait complete. 2024-05-21T17:21:39.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7f69c80041e0 con 0x7f69d00a70c0 2024-05-21T17:21:39.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69d7da4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.3:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f69b8000f90 con 0x7f69d00a70c0 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f69c8045d60 con 0x7f69d00a70c0 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: monmap: 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.203:6804/1468009857 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69b40436f0 con 0x7f69b403fa80 2024-05-21T17:21:39.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (unknown 2942983318 0 0) 0x7f69c8046120 con 0x7f69d00a70c0 2024-05-21T17:21:39.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69c67fc640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-05-21T17:21:39.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient: shutdown 2024-05-21T17:21:39.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69d5b40640 10 monclient: get_auth_request con 0x7f69b4038e80 auth_method 0 2024-05-21T17:21:39.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- 172.21.15.3:0/1289398666 >> v1:172.21.15.3:6793/0 conn(0x7f1814135bf0 legacy=0x7f18140a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- 172.21.15.3:0/1289398666 shutdown_connections 2024-05-21T17:21:39.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- 172.21.15.3:0/1289398666 >> 172.21.15.3:0/1289398666 conn(0x7f181406c8d0 msgr2=0x7f181406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- 172.21.15.3:0/1289398666 shutdown_connections 2024-05-21T17:21:39.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- 172.21.15.3:0/1289398666 wait complete. 2024-05-21T17:21:39.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 Processor -- start 2024-05-21T17:21:39.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f69d533f640 10 monclient: get_auth_request con 0x7f69b403fa80 auth_method 0 2024-05-21T17:21:39.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- start start 2024-05-21T17:21:39.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: init 2024-05-21T17:21:39.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient: monmap: 2024-05-21T17:21:39.529 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.530 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.531 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: _add_conns ranks=[1,6,4,3,7,5,8,0,2] 2024-05-21T17:21:39.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient(hunting): picked mon.noname-f con 0x7f91400a5bd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient(hunting): picked mon.noname-d con 0x7f91400a66c0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:39.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient(hunting): picked mon.noname-c con 0x7f91400a70c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:39.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 Processor -- start 2024-05-21T17:21:39.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9140141970 con 0x7f91400a5bd0 2024-05-21T17:21:39.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91400a79f0 con 0x7f91400a70c0 2024-05-21T17:21:39.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9140122180 con 0x7f91400a66c0 2024-05-21T17:21:39.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient(hunting): authenticate will time out at 966.622319s 2024-05-21T17:21:39.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f913f7fe640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f91400a70c0 0x7f91401d5dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47528/0 (socket says 172.21.15.3:47528) 2024-05-21T17:21:39.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f913e7fc640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f91400a66c0 0x7f91401d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:45944/0 (socket says 172.21.15.3:45944) 2024-05-21T17:21:39.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f913f7fe640 1 -- 172.21.15.3:0/3648226641 learned_addr learned my addr 172.21.15.3:0/3648226641 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- start start 2024-05-21T17:21:39.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient: init 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1848666419 0 0) 0x7f91400a79f0 con 0x7f91400a70c0 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): my global_id is 4288 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f912c003530 con 0x7f91400a70c0 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4213976337 0 0) 0x7f9140122180 con 0x7f91400a66c0 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): my global_id is 4236 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91400a79f0 con 0x7f91400a66c0 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127272424 0 0) 0x7f9140141970 con 0x7f91400a5bd0 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): my global_id is 4267 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9140122180 con 0x7f91400a5bd0 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 563909735 0 0) 0x7f91400a79f0 con 0x7f91400a66c0 2024-05-21T17:21:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient: _add_conns ranks=[3,7,5,6,0,1,8,4,2] 2024-05-21T17:21:39.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9140141970 con 0x7f91400a66c0 2024-05-21T17:21:39.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9134001a20 con 0x7f91400a66c0 2024-05-21T17:21:39.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient(hunting): picked mon.noname-g con 0x7f1814135bf0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): dump: 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient(hunting): picked mon.noname-i con 0x7f1814135fd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient(hunting): picked mon.noname-h con 0x7f181413fc90 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1814138e50 con 0x7f1814135bf0 2024-05-21T17:21:39.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f181409e510 con 0x7f181413fc90 2024-05-21T17:21:39.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18140a4440 con 0x7f1814135fd0 2024-05-21T17:21:39.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f1819029640 10 monclient(hunting): authenticate will time out at 966.623271s 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2134648019 0 0) 0x7f912c003530 con 0x7f91400a70c0 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91400a79f0 con 0x7f91400a70c0 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9138001d40 con 0x7f91400a70c0 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient(hunting): dump: 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.540 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2423812884 0 0) 0x7f9140141970 con 0x7f91400a66c0 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 >> v1:172.21.15.3:6791/0 conn(0x7f91400a70c0 legacy=0x7f91401d5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 >> v1:172.21.15.203:6789/0 conn(0x7f91400a5bd0 legacy=0x7f914013b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 monclient: found mon.d 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9140122460 con 0x7f91400a66c0 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 5 monclient: authenticate success, global_id 4236 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: _renew_subs 2024-05-21T17:21:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f181359e640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f181413fc90 0x7f18141d5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.3:58196/0 (socket says 172.21.15.3:58196) 2024-05-21T17:21:39.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f181359e640 1 -- 172.21.15.3:0/1991026800 learned_addr learned my addr 172.21.15.3:0/1991026800 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9134001d20 con 0x7f91400a66c0 2024-05-21T17:21:39.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f914013b7d0 con 0x7f91400a66c0 2024-05-21T17:21:39.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 --> v1:172.21.15.203:6800/4020495055 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fab4c06edf0 con 0x7fab2c03f940 2024-05-21T17:21:39.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9134003c10 con 0x7f91400a66c0 2024-05-21T17:21:39.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:39.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91277fe640 10 monclient: dump: 2024-05-21T17:21:39.544 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.544 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.545 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f913d7fa640 4 set_mon_vals no callback set 2024-05-21T17:21:39.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: _renew_subs 2024-05-21T17:21:39.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.460+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f91401424d0 con 0x7f91400a66c0 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012450583 0 0) 0x7f1814138e50 con 0x7f1814135bf0 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): my global_id is 4278 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1800003530 con 0x7f1814135bf0 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830533198 0 0) 0x7f18140a4440 con 0x7f1814135fd0 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): my global_id is 4318 2024-05-21T17:21:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1814138e50 con 0x7f1814135fd0 2024-05-21T17:21:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7f9134003db0 con 0x7f91400a66c0 2024-05-21T17:21:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab437fe640 1 -- 172.21.15.3:0/1009608127 <== osd.4 v1:172.21.15.203:6800/4020495055 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1800444376) 0x7fab4c06edf0 con 0x7fab2c03f940 2024-05-21T17:21:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f91457eb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f914006bf50 con 0x7f91400a66c0 2024-05-21T17:21:39.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== osd.3 v1:172.21.15.203:6804/1468009857 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69b40436f0 con 0x7f69b403fa80 2024-05-21T17:21:39.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f9134045a80 con 0x7f91400a66c0 2024-05-21T17:21:39.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6800/857120552 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f912c043c60 con 0x7f912c03fff0 2024-05-21T17:21:39.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== mon.6 v1:172.21.15.3:6792/0 9 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (unknown 2942983318 0 0) 0x7f9134045e30 con 0x7f91400a66c0 2024-05-21T17:21:39.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f91277fe640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-05-21T17:21:39.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f913effd640 10 monclient: get_auth_request con 0x7f912c039490 auth_method 0 2024-05-21T17:21:39.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 >> v1:172.21.15.203:6800/4020495055 conn(0x7fab2c03f940 legacy=0x7fab2c041d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f913f7fe640 10 monclient: get_auth_request con 0x7f912c03fff0 auth_method 0 2024-05-21T17:21:39.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 >> v1:172.21.15.203:6812/343827626 conn(0x7fab2c038d40 legacy=0x7fab2c03b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 10 monclient: shutdown 2024-05-21T17:21:39.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 >> v1:172.21.15.3:6793/0 conn(0x7fab4c0a0ea0 legacy=0x7fab4c141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab4ad9d640 1 -- 172.21.15.3:0/1009608127 reap_dead start 2024-05-21T17:21:39.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 shutdown_connections 2024-05-21T17:21:39.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 >> 172.21.15.3:0/1009608127 conn(0x7fab4c0a72c0 msgr2=0x7fab4c0a15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== osd.0 v1:172.21.15.3:6800/857120552 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f912c043c60 con 0x7f912c03fff0 2024-05-21T17:21:39.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 shutdown_connections 2024-05-21T17:21:39.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7fab50fc9640 1 -- 172.21.15.3:0/1009608127 wait complete. 2024-05-21T17:21:39.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3374977171 0 0) 0x7f1800003530 con 0x7f1814135bf0 2024-05-21T17:21:39.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18140a4440 con 0x7f1814135bf0 2024-05-21T17:21:39.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1804001840 con 0x7f1814135bf0 2024-05-21T17:21:39.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-05-21T17:21:39.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:39.553 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.553 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.554 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2134773425 0 0) 0x7f1814138e50 con 0x7f1814135fd0 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1800003530 con 0x7f1814135fd0 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1808001ae0 con 0x7f1814135fd0 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.555 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316045295 0 0) 0x7f181409e510 con 0x7f181413fc90 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient(hunting): my global_id is 4379 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1814138e50 con 0x7f181413fc90 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4010227160 0 0) 0x7f18140a4440 con 0x7f1814135bf0 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 >> v1:172.21.15.203:6792/0 conn(0x7f1814135fd0 legacy=0x7f18141d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 >> v1:172.21.15.203:6791/0 conn(0x7f181413fc90 legacy=0x7f18141d5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 monclient: found mon.g 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18141396f0 con 0x7f1814135bf0 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 5 monclient: authenticate success, global_id 4278 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 10 monclient: _renew_subs 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1814137860 con 0x7f1814135bf0 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 10 monclient: _renew_subs 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1814138160 con 0x7f1814135bf0 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18040019e0 con 0x7f1814135bf0 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1804004870 con 0x7f1814135bf0 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: dump: 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.559 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f181159a640 4 set_mon_vals no callback set 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f17f4000f90 con 0x7f1814135bf0 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 7 ==== mgrmap(e 4) ==== 44955+0+0 (unknown 3771251618 0 0) 0x7f18040101e0 con 0x7f1814135bf0 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f18040459b0 con 0x7f1814135bf0 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6808/2732955435 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1800043b10 con 0x7f180003ff40 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== mon.3 v1:172.21.15.203:6790/0 9 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (unknown 2942983318 0 0) 0x7f1804045d30 con 0x7f1814135bf0 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f17fb7fe640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f181259c640 10 monclient: get_auth_request con 0x7f18000393e0 auth_method 0 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.464+0000 7f181359e640 10 monclient: get_auth_request con 0x7f180003ff40 auth_method 0 2024-05-21T17:21:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.468+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== osd.5 v1:172.21.15.203:6808/2732955435 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1800043b10 con 0x7f180003ff40 2024-05-21T17:21:39.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 --> v1:172.21.15.3:6800/857120552 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f914006bf50 con 0x7f912c03fff0 2024-05-21T17:21:39.561 INFO:teuthology.orchestra.run.smithi003.stdout:47244640259 2024-05-21T17:21:39.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-21T17:21:39.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91277fe640 1 -- 172.21.15.3:0/3648226641 <== osd.0 v1:172.21.15.3:6800/857120552 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7f914006bf50 con 0x7f912c03fff0 2024-05-21T17:21:39.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 >> v1:172.21.15.3:6800/857120552 conn(0x7f912c03fff0 legacy=0x7f912c042410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 >> v1:172.21.15.203:6812/343827626 conn(0x7f912c039490 legacy=0x7f912c03b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 10 monclient: shutdown 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 >> v1:172.21.15.3:6792/0 conn(0x7f91400a66c0 legacy=0x7f91401d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f913f7fe640 1 -- 172.21.15.3:0/3648226641 reap_dead start 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 shutdown_connections 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 >> 172.21.15.3:0/3648226641 conn(0x7f91400ab7d0 msgr2=0x7f9140075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 shutdown_connections 2024-05-21T17:21:39.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.480+0000 7f91457eb640 1 -- 172.21.15.3:0/3648226641 wait complete. 2024-05-21T17:21:39.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.488+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 --> v1:172.21.15.203:6804/1468009857 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f69b8002c80 con 0x7f69b403fa80 2024-05-21T17:21:39.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.488+0000 7f69c67fc640 1 -- 172.21.15.3:0/1952714645 <== osd.3 v1:172.21.15.203:6804/1468009857 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 2570787547) 0x7f69b8002c80 con 0x7f69b403fa80 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 >> v1:172.21.15.203:6804/1468009857 conn(0x7f69b403fa80 legacy=0x7f69b4041ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 >> v1:172.21.15.203:6812/343827626 conn(0x7f69b4038e80 legacy=0x7f69b403b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 10 monclient: shutdown 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 >> v1:172.21.15.3:6790/0 conn(0x7f69d00a70c0 legacy=0x7f69d01d5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d6341640 1 -- 172.21.15.3:0/1952714645 reap_dead start 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 shutdown_connections 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 >> 172.21.15.3:0/1952714645 conn(0x7f69d00ab7d0 msgr2=0x7f69d009fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 shutdown_connections 2024-05-21T17:21:39.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.492+0000 7f69d7da4640 1 -- 172.21.15.3:0/1952714645 wait complete. 2024-05-21T17:21:39.567 INFO:teuthology.orchestra.run.smithi003.stdout:47244640260 2024-05-21T17:21:39.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-21T17:21:39.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 --> v1:172.21.15.203:6808/2732955435 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f17f4002c80 con 0x7f180003ff40 2024-05-21T17:21:39.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f17fb7fe640 1 -- 172.21.15.3:0/1991026800 <== osd.5 v1:172.21.15.203:6808/2732955435 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7f17f4002c80 con 0x7f180003ff40 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 >> v1:172.21.15.203:6808/2732955435 conn(0x7f180003ff40 legacy=0x7f1800042360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 >> v1:172.21.15.203:6812/343827626 conn(0x7f18000393e0 legacy=0x7f180003b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 10 monclient: shutdown 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 >> v1:172.21.15.203:6790/0 conn(0x7f1814135bf0 legacy=0x7f181413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f181359e640 1 -- 172.21.15.3:0/1991026800 reap_dead start 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 shutdown_connections 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 >> 172.21.15.3:0/1991026800 conn(0x7f181406c8d0 msgr2=0x7f1814130b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 shutdown_connections 2024-05-21T17:21:39.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.496+0000 7f1819029640 1 -- 172.21.15.3:0/1991026800 wait complete. 2024-05-21T17:21:39.570 INFO:teuthology.orchestra.run.smithi003.stdout:47244640258 2024-05-21T17:21:39.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-21T17:21:39.571 INFO:teuthology.orchestra.run.smithi003.stdout:47244640260 2024-05-21T17:21:39.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-21T17:21:39.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient: monmap: 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.685176+0000 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.685176+0000 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.685 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.686 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.686 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.686 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.686 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.686 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.686 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.686 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.686 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 1 Processor -- start 2024-05-21T17:21:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 1 -- start start 2024-05-21T17:21:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient: init 2024-05-21T17:21:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient: _add_conns ranks=[6,1,7,5,4,3,0,2,8] 2024-05-21T17:21:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient(hunting): picked mon.noname-d con 0x7f450c0a4a00 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient(hunting): picked mon.noname-f con 0x7f450c0a3530 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient(hunting): picked mon.noname-i con 0x7f450c0a3910 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c076740 con 0x7f450c0a3530 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c0a0fb0 con 0x7f450c0a4a00 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c09ecb0 con 0x7f450c0a3910 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450b7fe640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f450c0a4a00 0x7f450c0a2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:45960/0 (socket says 172.21.15.3:45960) 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450b7fe640 1 -- 172.21.15.3:0/2549660707 learned_addr learned my addr 172.21.15.3:0/2549660707 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f4511f9f640 10 monclient(hunting): authenticate will time out at 966.843215s 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 992364893 0 0) 0x7f450c0a0fb0 con 0x7f450c0a4a00 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient(hunting): my global_id is 4245 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f4003530 con 0x7f450c0a4a00 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652175759 0 0) 0x7f450c076740 con 0x7f450c0a3530 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient(hunting): my global_id is 4276 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f450c0a0fb0 con 0x7f450c0a3530 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3513124643 0 0) 0x7f44f4003530 con 0x7f450c0a4a00 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 >> v1:172.21.15.203:6792/0 conn(0x7f450c0a3910 legacy=0x7f450c13f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 >> v1:172.21.15.203:6789/0 conn(0x7f450c0a3530 legacy=0x7f450c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 monclient: found mon.noname-d 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c09dcd0 con 0x7f450c0a4a00 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f44fc001840 con 0x7f450c0a4a00 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.680+0000 7f450a7fc640 10 monclient: dump: 2024-05-21T17:21:39.691 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.692 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f44fc0019e0 con 0x7f450c0a4a00 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f450a7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f450a7fc640 1 -- 172.21.15.3:0/2549660707 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f44fc004bf0 con 0x7f450c0a4a00 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f450a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f450a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f450a7fc640 10 monclient: dump: 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.693 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 5 monclient: authenticate success, global_id 4245 2024-05-21T17:21:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 4 set_mon_vals no callback set 2024-05-21T17:21:39.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 10 monclient: shutdown 2024-05-21T17:21:39.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 1 -- 172.21.15.3:0/2549660707 >> v1:172.21.15.3:6792/0 conn(0x7f450c0a4a00 legacy=0x7f450c0a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 1 -- 172.21.15.3:0/2549660707 shutdown_connections 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 1 -- 172.21.15.3:0/2549660707 >> 172.21.15.3:0/2549660707 conn(0x7f450c0a3f10 msgr2=0x7f450c0a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 1 -- 172.21.15.3:0/2549660707 shutdown_connections 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 1 -- 172.21.15.3:0/2549660707 wait complete. 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 10 monclient: monmap: 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 1 Processor -- start 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 1 -- start start 2024-05-21T17:21:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 10 monclient: init 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.684+0000 7f4511f9f640 10 monclient: _add_conns ranks=[6,2,3,1,8,0,7,4,5] 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient(hunting): picked mon.noname-d con 0x7f450c0a3530 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f450b7fe640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f450c0a3530 0x7f450c136830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:45976/0 (socket says 172.21.15.3:45976) 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f450c0a3910 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f450b7fe640 1 -- 172.21.15.3:0/3627298889 learned_addr learned my addr 172.21.15.3:0/3627298889 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient(hunting): picked mon.noname-g con 0x7f450c0a4a00 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c073090 con 0x7f450c0a3910 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c096600 con 0x7f450c0a4a00 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c096040 con 0x7f450c0a3530 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient(hunting): authenticate will time out at 966.848702s 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539372394 0 0) 0x7f450c073090 con 0x7f450c0a3910 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): my global_id is 4259 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f8003530 con 0x7f450c0a3910 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265360561 0 0) 0x7f450c096040 con 0x7f450c0a3530 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): my global_id is 4254 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f450c073090 con 0x7f450c0a3530 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1040332225 0 0) 0x7f450c096600 con 0x7f450c0a4a00 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): my global_id is 4287 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f450c096040 con 0x7f450c0a4a00 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 81103930 0 0) 0x7f44f8003530 con 0x7f450c0a3910 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f450c096600 con 0x7f450c0a3910 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2609292245 0 0) 0x7f450c073090 con 0x7f450c0a3530 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44f8003530 con 0x7f450c0a3530 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4500001ae0 con 0x7f450c0a3910 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.702 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f44fc001840 con 0x7f450c0a3530 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 908463756 0 0) 0x7f450c096600 con 0x7f450c0a3910 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 >> v1:172.21.15.3:6792/0 conn(0x7f450c0a3530 legacy=0x7f450c136830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 >> v1:172.21.15.203:6790/0 conn(0x7f450c0a4a00 legacy=0x7f450c1d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 monclient: found mon.b 2024-05-21T17:21:39.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c096320 con 0x7f450c0a3910 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 5 monclient: authenticate success, global_id 4259 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient: _renew_subs 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f450c0732d0 con 0x7f450c0a3910 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient: _renew_subs 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f450c0723f0 con 0x7f450c0a3910 2024-05-21T17:21:39.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f45000038d0 con 0x7f450c0a3910 2024-05-21T17:21:39.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4509ffb640 4 set_mon_vals no callback set 2024-05-21T17:21:39.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4500001c80 con 0x7f450c0a3910 2024-05-21T17:21:39.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.688+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44ec005090 con 0x7f450c0a3910 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.692+0000 7f44f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.692+0000 7f44f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.692+0000 7f44f3fff640 10 monclient: dump: 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.692+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f450000ede0 con 0x7f450c0a3910 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.692+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f45000448f0 con 0x7f450c0a3910 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.696+0000 7f450b7fe640 10 monclient: get_auth_request con 0x7f44f80394d0 auth_method 0 2024-05-21T17:21:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.700+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f4500018700 con 0x7f450c0a3910 2024-05-21T17:21:39.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.700+0000 7f44f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.700+0000 7f44f3fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:39.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.712+0000 7efe0edfa640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.712+0000 7efe0edfa640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.712+0000 7efe0edfa640 10 monclient: monmap: 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.720070+0000 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.720070+0000 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.721 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.721 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.721 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.721 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.721 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.721 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.721 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient: monmap: 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.720523+0000 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.720523+0000 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.722 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.723 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.723 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.723 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.723 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.723 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 Processor -- start 2024-05-21T17:21:39.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- start start 2024-05-21T17:21:39.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient: init 2024-05-21T17:21:39.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 1 Processor -- start 2024-05-21T17:21:39.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 1 -- start start 2024-05-21T17:21:39.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient: init 2024-05-21T17:21:39.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient: _add_conns ranks=[1,2,4,6,7,8,0,5,3] 2024-05-21T17:21:39.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient(hunting): picked mon.noname-f con 0x7efe080a5bd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient(hunting): picked mon.noname-b con 0x7efe080a66c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient(hunting): picked mon.noname-c con 0x7efe080a70c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:39.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0809e480 con 0x7efe080a5bd0 2024-05-21T17:21:39.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0809cce0 con 0x7efe080a66c0 2024-05-21T17:21:39.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0809cfc0 con 0x7efe080a70c0 2024-05-21T17:21:39.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient(hunting): authenticate will time out at 966.877452s 2024-05-21T17:21:39.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe07fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7efe080a66c0 0x7efe0813dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47154/0 (socket says 172.21.15.3:47154) 2024-05-21T17:21:39.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe07fff640 1 -- 172.21.15.3:0/195438204 learned_addr learned my addr 172.21.15.3:0/195438204 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient: _add_conns ranks=[1,4,5,6,7,0,2,8,3] 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient(hunting): picked mon.noname-f con 0x7f6ac8135c80 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac80a3d50 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient(hunting): picked mon.noname-h con 0x7f6ac8136060 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac80a0240 con 0x7f6ac8135c80 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac81364d0 con 0x7f6ac80a3d50 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3499237329 0 0) 0x7efe0809cce0 con 0x7efe080a66c0 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient(hunting): my global_id is 4268 2024-05-21T17:21:39.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac80a4130 con 0x7f6ac8136060 2024-05-21T17:21:39.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdf8003530 con 0x7efe080a66c0 2024-05-21T17:21:39.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac7fff640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f6ac80a3d50 0x7f6ac813a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47534/0 (socket says 172.21.15.3:47534) 2024-05-21T17:21:39.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac7fff640 1 -- 172.21.15.3:0/4198078961 learned_addr learned my addr 172.21.15.3:0/4198078961 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient(hunting): authenticate will time out at 966.878211s 2024-05-21T17:21:39.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509435047 0 0) 0x7efe0809cfc0 con 0x7efe080a70c0 2024-05-21T17:21:39.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient(hunting): my global_id is 4297 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe0809cce0 con 0x7efe080a70c0 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1880188092 0 0) 0x7efdf8003530 con 0x7efe080a66c0 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 >> v1:172.21.15.3:6791/0 conn(0x7efe080a70c0 legacy=0x7efe08141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 >> v1:172.21.15.203:6789/0 conn(0x7efe080a5bd0 legacy=0x7efe080a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 monclient: found mon.noname-b 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe080742e0 con 0x7efe080a66c0 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efdf4001770 con 0x7efe080a66c0 2024-05-21T17:21:39.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 5 monclient: authenticate success, global_id 4268 2024-05-21T17:21:39.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: dump: 2024-05-21T17:21:39.733 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.735 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 752190286 0 0) 0x7f6ab4001050 con 0x7f6ac80a3d50 2024-05-21T17:21:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): my global_id is 4306 2024-05-21T17:21:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ab8005740 con 0x7f6ac80a3d50 2024-05-21T17:21:39.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1687479169 0 0) 0x7f6ac80a4130 con 0x7f6ac8136060 2024-05-21T17:21:39.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): my global_id is 4388 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ab4001050 con 0x7f6ac8136060 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792818685 0 0) 0x7f6ac80a0240 con 0x7f6ac8135c80 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient(hunting): my global_id is 4294 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ac80a4130 con 0x7f6ac8135c80 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efdf4001910 con 0x7efe080a66c0 2024-05-21T17:21:39.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 1 -- 172.21.15.3:0/195438204 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efdf4004bf0 con 0x7efe080a66c0 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 4 set_mon_vals no callback set 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe077fe640 10 monclient: dump: 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.739 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2813419459 0 0) 0x7f6ab8005740 con 0x7f6ac80a3d50 2024-05-21T17:21:39.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 >> v1:172.21.15.203:6791/0 conn(0x7f6ac8136060 legacy=0x7f6ac813e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 >> v1:172.21.15.203:6789/0 conn(0x7f6ac8135c80 legacy=0x7f6ac809fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 monclient: found mon.noname-c 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac809e4b0 con 0x7f6ac80a3d50 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6ab4001940 con 0x7f6ac80a3d50 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: dump: 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 5 monclient: authenticate success, global_id 4306 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ab4001ae0 con 0x7f6ac80a3d50 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 1 -- 172.21.15.3:0/4198078961 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6ab4004bf0 con 0x7f6ac80a3d50 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6ac77fe640 10 monclient: dump: 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.744 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7f6acf1e1640 4 set_mon_vals no callback set 2024-05-21T17:21:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 10 monclient: shutdown 2024-05-21T17:21:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- 172.21.15.3:0/195438204 >> v1:172.21.15.3:6790/0 conn(0x7efe080a66c0 legacy=0x7efe0813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- 172.21.15.3:0/195438204 shutdown_connections 2024-05-21T17:21:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- 172.21.15.3:0/195438204 >> 172.21.15.3:0/195438204 conn(0x7efe080ab7d0 msgr2=0x7efe080a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- 172.21.15.3:0/195438204 shutdown_connections 2024-05-21T17:21:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.716+0000 7efe0edfa640 1 -- 172.21.15.3:0/195438204 wait complete. 2024-05-21T17:21:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient: shutdown 2024-05-21T17:21:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- 172.21.15.3:0/4198078961 >> v1:172.21.15.3:6791/0 conn(0x7f6ac80a3d50 legacy=0x7f6ac813a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- 172.21.15.3:0/4198078961 shutdown_connections 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- 172.21.15.3:0/4198078961 >> 172.21.15.3:0/4198078961 conn(0x7f6ac80a8270 msgr2=0x7f6ac80a8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- 172.21.15.3:0/4198078961 shutdown_connections 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- 172.21.15.3:0/4198078961 wait complete. 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient: monmap: 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.748 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 1 Processor -- start 2024-05-21T17:21:39.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient: monmap: 2024-05-21T17:21:39.749 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.749 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.749 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.750 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 1 -- start start 2024-05-21T17:21:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient: init 2024-05-21T17:21:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 Processor -- start 2024-05-21T17:21:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- start start 2024-05-21T17:21:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient: init 2024-05-21T17:21:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient: _add_conns ranks=[3,2,5,1,0,8,7,6,4] 2024-05-21T17:21:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient(hunting): picked mon.noname-g con 0x7efe080a5bd0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient(hunting): picked mon.noname-b con 0x7efe080a66c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe07fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7efe080a66c0 0x7efe081cd240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47160/0 (socket says 172.21.15.3:47160) 2024-05-21T17:21:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe07fff640 1 -- 172.21.15.3:0/1173783232 learned_addr learned my addr 172.21.15.3:0/1173783232 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient(hunting): picked mon.noname-h con 0x7efe080a70c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient: _add_conns ranks=[3,2,5,6,8,1,4,0,7] 2024-05-21T17:21:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0809e6a0 con 0x7efe080a66c0 2024-05-21T17:21:39.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient(hunting): picked mon.noname-g con 0x7f6ac80a3d50 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:39.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac8135c80 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient(hunting): picked mon.noname-h con 0x7f6ac8136060 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac8120720 con 0x7f6ac8135c80 2024-05-21T17:21:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe081239b0 con 0x7efe080a5bd0 2024-05-21T17:21:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe08123110 con 0x7efe080a70c0 2024-05-21T17:21:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7efe0edfa640 10 monclient(hunting): authenticate will time out at 966.882826s 2024-05-21T17:21:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac7fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6ac8135c80 0x7f6ac81ce6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47172/0 (socket says 172.21.15.3:47172) 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac7fff640 1 -- 172.21.15.3:0/2134208503 learned_addr learned my addr 172.21.15.3:0/2134208503 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac811f270 con 0x7f6ac80a3d50 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac811ebb0 con 0x7f6ac8136060 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6acf1e1640 10 monclient(hunting): authenticate will time out at 966.882919s 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1556957537 0 0) 0x7f6ac8120720 con 0x7f6ac8135c80 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): my global_id is 4286 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6aac003530 con 0x7f6ac8135c80 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3459043826 0 0) 0x7f6ac811ebb0 con 0x7f6ac8136060 2024-05-21T17:21:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): my global_id is 4406 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ac8120720 con 0x7f6ac8136060 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 379030294 0 0) 0x7f6ac811f270 con 0x7f6ac80a3d50 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): my global_id is 4305 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ac811ebb0 con 0x7f6ac80a3d50 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2564671977 0 0) 0x7f6aac003530 con 0x7f6ac8135c80 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ac811f270 con 0x7f6ac8135c80 2024-05-21T17:21:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6ab4004b30 con 0x7f6ac8135c80 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): dump: 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1320261682 0 0) 0x7f6ac811f270 con 0x7f6ac8135c80 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.720+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 >> v1:172.21.15.203:6791/0 conn(0x7f6ac8136060 legacy=0x7f6ac81d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 >> v1:172.21.15.203:6790/0 conn(0x7f6ac80a3d50 legacy=0x7f6ac81c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 1 monclient: found mon.b 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac8120a00 con 0x7f6ac8135c80 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 5 monclient: authenticate success, global_id 4286 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 10 monclient: _renew_subs 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac809d7e0 con 0x7f6ac8135c80 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 10 monclient: _renew_subs 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ac80733e0 con 0x7f6ac8135c80 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ab4001cd0 con 0x7f6ac8135c80 2024-05-21T17:21:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6ab40051a0 con 0x7f6ac8135c80 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac6ffd640 4 set_mon_vals no callback set 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 10 monclient: dump: 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa8005090 con 0x7f6ac8135c80 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f6ab4010320 con 0x7f6ac8135c80 2024-05-21T17:21:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f6ab4045d90 con 0x7f6ac8135c80 2024-05-21T17:21:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.724+0000 7f6accf7d640 10 monclient: get_auth_request con 0x7f6aac038ab0 auth_method 0 2024-05-21T17:21:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f6ac81364d0 con 0x7f6ac8135c80 2024-05-21T17:21:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7f6ac57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7f6ac57fa640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 100972771 0 0) 0x7efe0809e6a0 con 0x7efe080a66c0 2024-05-21T17:21:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): my global_id is 4277 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdec003530 con 0x7efe080a66c0 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 944843577 0 0) 0x7efe08123110 con 0x7efe080a70c0 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): my global_id is 4397 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe0809e6a0 con 0x7efe080a70c0 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706411307 0 0) 0x7efe081239b0 con 0x7efe080a5bd0 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): my global_id is 4296 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe08123110 con 0x7efe080a5bd0 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3114617038 0 0) 0x7efdec003530 con 0x7efe080a66c0 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe081239b0 con 0x7efe080a66c0 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efdf40016b0 con 0x7efe080a66c0 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): dump: 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.728+0000 7efe057fa640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2576163537 0 0) 0x7efe0809e6a0 con 0x7efe080a70c0 2024-05-21T17:21:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdec003530 con 0x7efe080a70c0 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4124550359 0 0) 0x7efe08123110 con 0x7efe080a5bd0 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe0809e6a0 con 0x7efe080a5bd0 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3472945305 0 0) 0x7efe081239b0 con 0x7efe080a66c0 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 >> v1:172.21.15.203:6791/0 conn(0x7efe080a70c0 legacy=0x7efe081d55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 >> v1:172.21.15.203:6790/0 conn(0x7efe080a5bd0 legacy=0x7efe0813c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 monclient: found mon.b 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0813c740 con 0x7efe080a66c0 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 5 monclient: authenticate success, global_id 4277 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efdf4004b20 con 0x7efe080a66c0 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 10 monclient: _renew_subs 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe08121f70 con 0x7efe080a66c0 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efdf4005230 con 0x7efe080a66c0 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe057fa640 10 monclient: dump: 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe06ffd640 4 set_mon_vals no callback set 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 10 monclient: _renew_subs 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efe081cdb20 con 0x7efe080a66c0 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.732+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe0806bf50 con 0x7efe080a66c0 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.736+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7efdf40103b0 con 0x7efe080a66c0 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.736+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7efdf4045920 con 0x7efe080a66c0 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.736+0000 7efe0cb96640 10 monclient: get_auth_request con 0x7efdec039530 auth_method 0 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.740+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7efdf4015b60 con 0x7efe080a66c0 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.740+0000 7efe057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.740+0000 7efe057fa640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:39.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient: monmap: 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.764336+0000 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.764336+0000 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 1 Processor -- start 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 1 -- start start 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient: init 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient: _add_conns ranks=[6,1,7,8,5,2,4,3,0] 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient(hunting): picked mon.noname-d con 0x7ff1000a5bd0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient(hunting): picked mon.noname-f con 0x7ff1000a66c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient(hunting): picked mon.noname-i con 0x7ff1000a70c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10009e480 con 0x7ff1000a66c0 2024-05-21T17:21:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10009cce0 con 0x7ff1000a5bd0 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0fffff640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7ff1000a5bd0 0x7ff1000a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:45992/0 (socket says 172.21.15.3:45992) 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0fffff640 1 -- 172.21.15.3:0/3540895223 learned_addr learned my addr 172.21.15.3:0/3540895223 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 1 -- 172.21.15.3:0/3540895223 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10009cfc0 con 0x7ff1000a70c0 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient(hunting): authenticate will time out at 966.922339s 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1697245342 0 0) 0x7ff10009cce0 con 0x7ff1000a5bd0 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): my global_id is 4263 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e8003530 con 0x7ff1000a5bd0 2024-05-21T17:21:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346976081 0 0) 0x7ff10009e480 con 0x7ff1000a66c0 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): my global_id is 4303 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff10009cce0 con 0x7ff1000a66c0 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1010322366 0 0) 0x7ff10009cfc0 con 0x7ff1000a70c0 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient(hunting): my global_id is 4336 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff10009e480 con 0x7ff1000a70c0 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3422744369 0 0) 0x7ff0e8003530 con 0x7ff1000a5bd0 2024-05-21T17:21:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 >> v1:172.21.15.203:6792/0 conn(0x7ff1000a70c0 legacy=0x7ff100141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 >> v1:172.21.15.203:6789/0 conn(0x7ff1000a66c0 legacy=0x7ff10013dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 monclient: found mon.noname-d 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1000742e0 con 0x7ff1000a5bd0 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff0f00017c0 con 0x7ff1000a5bd0 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: dump: 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.788 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 5 monclient: authenticate success, global_id 4263 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0f0001960 con 0x7ff1000a5bd0 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 1 -- 172.21.15.3:0/3540895223 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff0f0004bf0 con 0x7ff1000a5bd0 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff0feffd640 10 monclient: dump: 2024-05-21T17:21:39.789 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.790 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.760+0000 7ff1062a3640 4 set_mon_vals no callback set 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient: shutdown 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- 172.21.15.3:0/3540895223 >> v1:172.21.15.3:6792/0 conn(0x7ff1000a5bd0 legacy=0x7ff1000a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- 172.21.15.3:0/3540895223 shutdown_connections 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- 172.21.15.3:0/3540895223 >> 172.21.15.3:0/3540895223 conn(0x7ff1000ab7d0 msgr2=0x7ff1000a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- 172.21.15.3:0/3540895223 shutdown_connections 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- 172.21.15.3:0/3540895223 wait complete. 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient: monmap: 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.792 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 Processor -- start 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- start start 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient: init 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient: _add_conns ranks=[7,4,2,1,6,8,3,5,0] 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient(hunting): picked mon.noname-i con 0x7ff1000a5bd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0ff7fe640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7ff1000a66c0 0x7ff10013dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47538/0 (socket says 172.21.15.3:47538) 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0ff7fe640 1 -- 172.21.15.3:0/3427350628 learned_addr learned my addr 172.21.15.3:0/3427350628 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient(hunting): picked mon.noname-c con 0x7ff1000a66c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient(hunting): picked mon.noname-b con 0x7ff1000a70c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10009e6a0 con 0x7ff1000a70c0 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff100141a50 con 0x7ff1000a66c0 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3849847981 0 0) 0x7ff100141a50 con 0x7ff1000a66c0 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff100141d30 con 0x7ff1000a5bd0 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff1062a3640 10 monclient(hunting): authenticate will time out at 966.927879s 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 10 monclient(hunting): my global_id is 4315 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ec003530 con 0x7ff1000a66c0 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201022169 0 0) 0x7ff10009e480 con 0x7ff1000a70c0 2024-05-21T17:21:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 10 monclient(hunting): my global_id is 4295 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.764+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff100141a50 con 0x7ff1000a70c0 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2829837186 0 0) 0x7ff100141a50 con 0x7ff1000a70c0 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff10009e480 con 0x7ff1000a70c0 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff0f8001bb0 con 0x7ff1000a70c0 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): dump: 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.795 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1274318252 0 0) 0x7ff100141d30 con 0x7ff1000a5bd0 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): my global_id is 4345 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff100141a50 con 0x7ff1000a5bd0 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 466527259 0 0) 0x7ff0ec003530 con 0x7ff1000a66c0 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff100141d30 con 0x7ff1000a66c0 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff0f4001c60 con 0x7ff1000a66c0 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient(hunting): dump: 2024-05-21T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3888773789 0 0) 0x7ff10009e480 con 0x7ff1000a70c0 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 >> v1:172.21.15.203:6792/0 conn(0x7ff1000a5bd0 legacy=0x7ff100142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 >> v1:172.21.15.3:6791/0 conn(0x7ff1000a66c0 legacy=0x7ff10013dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 monclient: found mon.b 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff100139620 con 0x7ff1000a70c0 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 5 monclient: authenticate success, global_id 4295 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 10 monclient: _renew_subs 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff10013fa40 con 0x7ff1000a70c0 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 10 monclient: _renew_subs 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1000a79d0 con 0x7ff1000a70c0 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0f80039a0 con 0x7ff1000a70c0 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff0f8003ed0 con 0x7ff1000a70c0 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fe7fc640 4 set_mon_vals no callback set 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 10 monclient: dump: 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.801 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e0005090 con 0x7ff1000a70c0 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7ff0f8004070 con 0x7ff1000a70c0 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7ff0f8044a90 con 0x7ff1000a70c0 2024-05-21T17:21:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7ff0fffff640 10 monclient: get_auth_request con 0x7ff0ec041d20 auth_method 0 2024-05-21T17:21:39.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7f0118ee2640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7f96498b2640 10 monclient: get_monmap_and_config 2024-05-21T17:21:39.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7f96498b2640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7f0118ee2640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7f96498b2640 10 monclient: monmap: 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.775916+0000 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.775916+0000 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.804 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.805 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.805 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.805 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.805 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.805 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.805 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.768+0000 7f0118ee2640 10 monclient: monmap: 2024-05-21T17:21:39.805 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:39.775787+0000 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:39.775787+0000 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 Processor -- start 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 -- start start 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient: init 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 1 Processor -- start 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 1 -- start start 2024-05-21T17:21:39.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient: init 2024-05-21T17:21:39.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient: _add_conns ranks=[0,8,6,4,7,5,3,2,1] 2024-05-21T17:21:39.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient(hunting): picked mon.noname-a con 0x7f010c02a550 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01137fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f010c02a550 0x7f010c022fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34914/0 (socket says 172.21.15.3:34914) 2024-05-21T17:21:39.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0112ffd640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f010c028270 0x7f010c028690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41672/0 (socket says 172.21.15.3:41672) 2024-05-21T17:21:39.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient(hunting): picked mon.noname-e con 0x7f010c028270 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:39.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient: _add_conns ranks=[8,7,3,6,5,4,2,0,1] 2024-05-21T17:21:39.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient(hunting): picked mon.noname-e con 0x7f96440a2820 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:39.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient(hunting): picked mon.noname-i con 0x7f96440a0ed0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient(hunting): picked mon.noname-d con 0x7f010c0c2bf0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:39.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0112ffd640 1 -- 172.21.15.3:0/3697056485 learned_addr learned my addr 172.21.15.3:0/3697056485 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient(hunting): picked mon.noname-g con 0x7f96440a12b0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:39.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 -- 172.21.15.3:0/3697056485 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c025a30 con 0x7f010c02a550 2024-05-21T17:21:39.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9642ffd640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f96440a2820 0x7f96440a2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41676/0 (socket says 172.21.15.3:41676) 2024-05-21T17:21:39.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9642ffd640 1 -- 172.21.15.3:0/473947185 learned_addr learned my addr 172.21.15.3:0/473947185 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 -- 172.21.15.3:0/3697056485 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c025190 con 0x7f010c0c2bf0 2024-05-21T17:21:39.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 1 -- 172.21.15.3:0/473947185 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96440a3310 con 0x7f96440a12b0 2024-05-21T17:21:39.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 -- 172.21.15.3:0/3697056485 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c025470 con 0x7f010c028270 2024-05-21T17:21:39.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 1 -- 172.21.15.3:0/473947185 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f964406df50 con 0x7f96440a0ed0 2024-05-21T17:21:39.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676581721 0 0) 0x7f010c025190 con 0x7f010c0c2bf0 2024-05-21T17:21:39.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 1 -- 172.21.15.3:0/473947185 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f964406e230 con 0x7f96440a2820 2024-05-21T17:21:39.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient(hunting): authenticate will time out at 966.934375s 2024-05-21T17:21:39.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient(hunting): authenticate will time out at 966.934432s 2024-05-21T17:21:39.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): my global_id is 4272 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0104003530 con 0x7f010c0c2bf0 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069027225 0 0) 0x7f010c025a30 con 0x7f010c02a550 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): my global_id is 4320 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f010c025190 con 0x7f010c02a550 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3360360073 0 0) 0x7f010c025470 con 0x7f010c028270 2024-05-21T17:21:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient(hunting): my global_id is 4292 2024-05-21T17:21:39.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f010c025a30 con 0x7f010c028270 2024-05-21T17:21:39.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4172066108 0 0) 0x7f96440a3310 con 0x7f96440a12b0 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): my global_id is 4314 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f962c003530 con 0x7f96440a12b0 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135898030 0 0) 0x7f964406e230 con 0x7f96440a2820 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): my global_id is 4301 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96440a3310 con 0x7f96440a2820 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 44797651 0 0) 0x7f964406df50 con 0x7f96440a0ed0 2024-05-21T17:21:39.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient(hunting): my global_id is 4354 2024-05-21T17:21:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f964406e230 con 0x7f96440a0ed0 2024-05-21T17:21:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1383443679 0 0) 0x7f010c025190 con 0x7f010c02a550 2024-05-21T17:21:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 >> v1:172.21.15.3:6793/0 conn(0x7f010c028270 legacy=0x7f010c028690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 >> v1:172.21.15.3:6792/0 conn(0x7f010c0c2bf0 legacy=0x7f010c0c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 monclient: found mon.noname-a 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c025750 con 0x7f010c02a550 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0108001840 con 0x7f010c02a550 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: dump: 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.818 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.819 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 5 monclient: authenticate success, global_id 4320 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01080019e0 con 0x7f010c02a550 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 1 -- 172.21.15.3:0/3697056485 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0108004b30 con 0x7f010c02a550 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 4 set_mon_vals no callback set 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f01127fc640 10 monclient: dump: 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.820 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.821 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1642996924 0 0) 0x7f962c003530 con 0x7f96440a12b0 2024-05-21T17:21:39.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 >> v1:172.21.15.3:6793/0 conn(0x7f96440a2820 legacy=0x7f96440a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 >> v1:172.21.15.203:6792/0 conn(0x7f96440a0ed0 legacy=0x7f964413dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 monclient: found mon.noname-g 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.203:6790/0 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 --> v1:172.21.15.203:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964406c7a0 con 0x7f96440a12b0 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f963c001bc0 con 0x7f96440a12b0 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: dump: 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.823 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 5 monclient: authenticate success, global_id 4314 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 <== mon.3 v1:172.21.15.203:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f963c001d60 con 0x7f96440a12b0 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 1 -- 172.21.15.3:0/473947185 <== mon.3 v1:172.21.15.203:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f963c003ce0 con 0x7f96440a12b0 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f9641ffb640 10 monclient: dump: 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 10 monclient: shutdown 2024-05-21T17:21:39.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 -- 172.21.15.3:0/3697056485 >> v1:172.21.15.3:6789/0 conn(0x7f010c02a550 legacy=0x7f010c022fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:39.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f96498b2640 4 set_mon_vals no callback set 2024-05-21T17:21:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 -- 172.21.15.3:0/3697056485 shutdown_connections 2024-05-21T17:21:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 -- 172.21.15.3:0/3697056485 >> 172.21.15.3:0/3697056485 conn(0x7f010c007550 msgr2=0x7f010c007960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.772+0000 7f0118ee2640 1 -- 172.21.15.3:0/3697056485 shutdown_connections 2024-05-21T17:21:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 1 -- 172.21.15.3:0/3697056485 wait complete. 2024-05-21T17:21:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: monmap: 2024-05-21T17:21:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.829 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 1 Processor -- start 2024-05-21T17:21:39.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 10 monclient: shutdown 2024-05-21T17:21:39.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 1 -- start start 2024-05-21T17:21:39.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: init 2024-05-21T17:21:39.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 1 -- 172.21.15.3:0/473947185 >> v1:172.21.15.203:6790/0 conn(0x7f96440a12b0 legacy=0x7f96441413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 1 -- 172.21.15.3:0/473947185 shutdown_connections 2024-05-21T17:21:39.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 1 -- 172.21.15.3:0/473947185 >> 172.21.15.3:0/473947185 conn(0x7f96440a9480 msgr2=0x7f96440a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 1 -- 172.21.15.3:0/473947185 shutdown_connections 2024-05-21T17:21:39.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 1 -- 172.21.15.3:0/473947185 wait complete. 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: _add_conns ranks=[5,7,8,2,0,6,3,1,4] 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient(hunting): picked mon.noname-h con 0x7f010c028270 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient(hunting): picked mon.noname-i con 0x7f010c02a550 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient(hunting): picked mon.noname-e con 0x7f010c0c2bf0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c01ffa0 con 0x7f010c028270 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c0bd380 con 0x7f010c02a550 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c01fcc0 con 0x7f010c0c2bf0 2024-05-21T17:21:39.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient(hunting): authenticate will time out at 966.937531s 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0113fff640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f010c0c2bf0 0x7f010c0bc6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41688/0 (socket says 172.21.15.3:41688) 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0113fff640 1 -- 172.21.15.3:0/3075146932 learned_addr learned my addr 172.21.15.3:0/3075146932 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 10 monclient: build_initial_monmap 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 10 monclient: monmap: 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:39.833 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:39.834 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978383510 0 0) 0x7f010c01fcc0 con 0x7f010c0c2bf0 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): my global_id is 4310 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00f8003530 con 0x7f010c0c2bf0 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 52030398 0 0) 0x7f010c025750 con 0x7f010c028270 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): my global_id is 4415 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f010c01fcc0 con 0x7f010c028270 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3958342891 0 0) 0x7f010c0bd380 con 0x7f010c02a550 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): my global_id is 4363 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f010c025750 con 0x7f010c02a550 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 275614833 0 0) 0x7f00f8003530 con 0x7f010c0c2bf0 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f010c0bd380 con 0x7f010c0c2bf0 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0114069890 con 0x7f010c0c2bf0 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:39.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1377805144 0 0) 0x7f010c01fcc0 con 0x7f010c028270 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f00f8003530 con 0x7f010c028270 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f01080013c0 con 0x7f010c028270 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1057658398 0 0) 0x7f010c025750 con 0x7f010c02a550 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f010c01fcc0 con 0x7f010c02a550 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0100001940 con 0x7f010c02a550 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 1 Processor -- start 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 1 -- start start 2024-05-21T17:21:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f96498b2640 10 monclient: init 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f96498b2640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f96498b2640 10 monclient: _add_conns ranks=[0,5,8,7,6,2,4,3,1] 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f96498b2640 10 monclient(hunting): picked mon.noname-a con 0x7f96440a0ed0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f96498b2640 10 monclient(hunting): picked mon.noname-h con 0x7f96440a12b0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f9642ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f96440a0ed0 0x7f964413cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34930/0 (socket says 172.21.15.3:34930) 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f96437fe640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f96440a2820 0x7f96441d5a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41694/0 (socket says 172.21.15.3:41694) 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f96427fc640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f96440a12b0 0x7f964413d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.3:58238/0 (socket says 172.21.15.3:58238) 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f9642ffd640 1 -- 172.21.15.3:0/1914958611 learned_addr learned my addr 172.21.15.3:0/1914958611 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f96498b2640 10 monclient(hunting): picked mon.noname-e con 0x7f96440a2820 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f96498b2640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9644140170 con 0x7f96440a0ed0 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7ff10009e6a0 con 0x7ff1000a70c0 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7ff0fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7ff0fcff9640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 840692362 0 0) 0x7f010c0bd380 con 0x7f010c0c2bf0 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 >> v1:172.21.15.203:6792/0 conn(0x7f010c02a550 legacy=0x7f010c0bbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 >> v1:172.21.15.203:6791/0 conn(0x7f010c028270 legacy=0x7f010c0bb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 monclient: found mon.e 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c01f900 con 0x7f010c0c2bf0 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f00fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 5 monclient: authenticate success, global_id 4310 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: _renew_subs 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f010c15aa70 con 0x7f010c0c2bf0 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: _renew_subs 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.776+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f010c15ae10 con 0x7f010c0c2bf0 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f0118ee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f0118ee2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f0118ee2640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010c0066b0 con 0x7f010c0c2bf0 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0114051780 con 0x7f010c0c2bf0 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f00fffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0114050d60 con 0x7f010c0c2bf0 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f00fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f00fffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f00fffff640 10 monclient: dump: 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f0111ffb640 4 set_mon_vals no callback set 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f011406b6c0 con 0x7f010c0c2bf0 2024-05-21T17:21:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f01140c2000 con 0x7f010c0c2bf0 2024-05-21T17:21:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.780+0000 7f01137fe640 10 monclient: get_auth_request con 0x7f00f80392d0 auth_method 0 2024-05-21T17:21:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1817718525 0 0) 0x7f9634001220 con 0x7f96440a0ed0 2024-05-21T17:21:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f011406eba0 con 0x7f010c0c2bf0 2024-05-21T17:21:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f00fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f00fffff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f96498b2640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f964413ddc0 con 0x7f96440a12b0 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f96498b2640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f964406d1f0 con 0x7f96440a2820 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f96498b2640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f96498b2640 10 monclient(hunting): authenticate will time out at 966.951699s 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 10 monclient(hunting): my global_id is 4329 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9630003530 con 0x7f96440a0ed0 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591002903 0 0) 0x7f964413ddc0 con 0x7f96440a12b0 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 10 monclient(hunting): my global_id is 4424 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.788+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9634001220 con 0x7f96440a12b0 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250829257 0 0) 0x7f964406d1f0 con 0x7f96440a2820 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): my global_id is 4319 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f964413ddc0 con 0x7f96440a2820 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3042680731 0 0) 0x7f9630003530 con 0x7f96440a0ed0 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f964406d1f0 con 0x7f96440a0ed0 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9634001cd0 con 0x7f96440a0ed0 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3633314822 0 0) 0x7f9634001220 con 0x7f96440a12b0 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9630003530 con 0x7f96440a12b0 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9638001560 con 0x7f96440a12b0 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient(hunting): dump: 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.854 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 574580861 0 0) 0x7f964406d1f0 con 0x7f96440a0ed0 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 >> v1:172.21.15.3:6793/0 conn(0x7f96440a2820 legacy=0x7f96441d5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 >> v1:172.21.15.203:6791/0 conn(0x7f96440a12b0 legacy=0x7f964413d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 monclient: found mon.a 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96440740e0 con 0x7f96440a0ed0 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient: _finish_auth 0 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96498b2640 5 monclient: authenticate success, global_id 4329 2024-05-21T17:21:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96498b2640 10 monclient: _renew_subs 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96498b2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96498b2640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f96440738f0 con 0x7f96440a0ed0 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96498b2640 10 monclient: _renew_subs 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96498b2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96498b2640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f964413fd50 con 0x7f96440a0ed0 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9634004aa0 con 0x7f96440a0ed0 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9634004fd0 con 0x7f96440a0ed0 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 10 monclient: dump: 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:39.856 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:39.857 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96417fa640 4 set_mon_vals no callback set 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f9629ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f9634005230 con 0x7f96440a0ed0 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f96340460b0 con 0x7f96440a0ed0 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f9629ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f9629ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f9629ffb640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9624005090 con 0x7f96440a0ed0 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.792+0000 7f96427fc640 10 monclient: get_auth_request con 0x7f9630039540 auth_method 0 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.804+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f9644140170 con 0x7f96440a0ed0 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.804+0000 7f962bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:39.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.804+0000 7f962bfff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:39.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-21T17:21:39.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-21T17:21:39.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7efe0806cb90 con 0x7efe080a66c0 2024-05-21T17:21:39.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe057fa640 1 -- 172.21.15.3:0/1173783232 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 555833778) 0x7efdf4015460 con 0x7efe080a66c0 2024-05-21T17:21:39.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-21T17:21:39.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe057fa640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:39.922 INFO:teuthology.orchestra.run.smithi003.stdout:47244640258 2024-05-21T17:21:39.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 >> v1:172.21.15.203:6812/343827626 conn(0x7efdec039530 legacy=0x7efdec03b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 10 monclient: shutdown 2024-05-21T17:21:39.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 >> v1:172.21.15.3:6790/0 conn(0x7efe080a66c0 legacy=0x7efe081cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 shutdown_connections 2024-05-21T17:21:39.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 >> 172.21.15.3:0/1173783232 conn(0x7efe080ab7d0 msgr2=0x7efe08075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 shutdown_connections 2024-05-21T17:21:39.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.916+0000 7efe0edfa640 1 -- 172.21.15.3:0/1173783232 wait complete. 2024-05-21T17:21:39.938 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-05-21T17:21:39.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-21T17:21:39.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-21T17:21:39.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:39.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff0e0005380 con 0x7ff1000a70c0 2024-05-21T17:21:39.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff0fcff9640 1 -- 172.21.15.3:0/3427350628 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 3327677263) 0x7ff0f8014bb0 con 0x7ff1000a70c0 2024-05-21T17:21:39.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff0fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-21T17:21:39.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff0fcff9640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:39.957 INFO:teuthology.orchestra.run.smithi003.stdout:47244640257 2024-05-21T17:21:39.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 >> v1:172.21.15.203:6812/343827626 conn(0x7ff0ec041d20 legacy=0x7ff0ec0441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 10 monclient: shutdown 2024-05-21T17:21:39.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:39.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 >> v1:172.21.15.3:6790/0 conn(0x7ff1000a70c0 legacy=0x7ff10013e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:39.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 shutdown_connections 2024-05-21T17:21:39.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 >> 172.21.15.3:0/3427350628 conn(0x7ff1000ab7d0 msgr2=0x7ff1000a9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:39.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 shutdown_connections 2024-05-21T17:21:39.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:39.952+0000 7ff1062a3640 1 -- 172.21.15.3:0/3427350628 wait complete. 2024-05-21T17:21:39.972 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.3 2024-05-21T17:21:40.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.024+0000 7f4511f9f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-21T17:21:40.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.024+0000 7f4511f9f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-21T17:21:40.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.024+0000 7f4511f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:40.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.024+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f44ec005380 con 0x7f450c0a3910 2024-05-21T17:21:40.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.024+0000 7f44f3fff640 1 -- 172.21.15.3:0/3627298889 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 555833778) 0x7f450001d130 con 0x7f450c0a3910 2024-05-21T17:21:40.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.024+0000 7f44f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-21T17:21:40.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.024+0000 7f44f3fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:40.030 INFO:teuthology.orchestra.run.smithi003.stdout:47244640258 2024-05-21T17:21:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.028+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 >> v1:172.21.15.203:6812/343827626 conn(0x7f44f80394d0 legacy=0x7f44f803b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.028+0000 7f4511f9f640 10 monclient: shutdown 2024-05-21T17:21:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.028+0000 7f4511f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.028+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 >> v1:172.21.15.3:6790/0 conn(0x7f450c0a3910 legacy=0x7f450c136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.028+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 shutdown_connections 2024-05-21T17:21:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.028+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 >> 172.21.15.3:0/3627298889 conn(0x7f450c0a3f10 msgr2=0x7f450c0943c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.028+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 shutdown_connections 2024-05-21T17:21:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.028+0000 7f4511f9f640 1 -- 172.21.15.3:0/3627298889 wait complete. 2024-05-21T17:21:40.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.032+0000 7f6acf1e1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-21T17:21:40.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.032+0000 7f6acf1e1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-21T17:21:40.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.032+0000 7f6acf1e1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:40.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.032+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f6aa8005380 con 0x7f6ac8135c80 2024-05-21T17:21:40.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6ac57fa640 1 -- 172.21.15.3:0/2134208503 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 555833778) 0x7f6ab401ce20 con 0x7f6ac8135c80 2024-05-21T17:21:40.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6ac57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-21T17:21:40.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6ac57fa640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:40.040 INFO:teuthology.orchestra.run.smithi003.stdout:47244640258 2024-05-21T17:21:40.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 >> v1:172.21.15.203:6812/343827626 conn(0x7f6aac038ab0 legacy=0x7f6aac03af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:40.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6acf1e1640 10 monclient: shutdown 2024-05-21T17:21:40.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6acf1e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:40.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 >> v1:172.21.15.3:6790/0 conn(0x7f6ac8135c80 legacy=0x7f6ac81ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:40.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 shutdown_connections 2024-05-21T17:21:40.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 >> 172.21.15.3:0/2134208503 conn(0x7f6ac80a8270 msgr2=0x7f6ac80a63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:40.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 shutdown_connections 2024-05-21T17:21:40.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.036+0000 7f6acf1e1640 1 -- 172.21.15.3:0/2134208503 wait complete. 2024-05-21T17:21:40.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-21T17:21:40.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-21T17:21:40.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:40.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f010c024530 con 0x7f010c0c2bf0 2024-05-21T17:21:40.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f00fffff640 1 -- 172.21.15.3:0/3075146932 <== mon.8 v1:172.21.15.3:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 847495621) 0x7f011406e4a0 con 0x7f010c0c2bf0 2024-05-21T17:21:40.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f00fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-21T17:21:40.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f00fffff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:40.045 INFO:teuthology.orchestra.run.smithi003.stdout:47244640259 2024-05-21T17:21:40.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 >> v1:172.21.15.203:6812/343827626 conn(0x7f00f80392d0 legacy=0x7f00f803b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:40.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 10 monclient: shutdown 2024-05-21T17:21:40.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:40.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 >> v1:172.21.15.3:6793/0 conn(0x7f010c0c2bf0 legacy=0x7f010c0bc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:40.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 shutdown_connections 2024-05-21T17:21:40.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 >> 172.21.15.3:0/3075146932 conn(0x7f010c007550 msgr2=0x7f010c02d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:40.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 shutdown_connections 2024-05-21T17:21:40.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.040+0000 7f0118ee2640 1 -- 172.21.15.3:0/3075146932 wait complete. 2024-05-21T17:21:40.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2024-05-21T17:21:40.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.048+0000 7f9629ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-21T17:21:40.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.048+0000 7f9629ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-21T17:21:40.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.048+0000 7f9629ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:40.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.048+0000 7f9629ffb640 1 -- 172.21.15.3:0/1914958611 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f9624005380 con 0x7f96440a0ed0 2024-05-21T17:21:40.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.048+0000 7f962bfff640 1 -- 172.21.15.3:0/1914958611 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 847495621) 0x7f9634016090 con 0x7f96440a0ed0 2024-05-21T17:21:40.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.048+0000 7f962bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-21T17:21:40.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.048+0000 7f962bfff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:40.056 INFO:teuthology.orchestra.run.smithi003.stdout:47244640259 2024-05-21T17:21:40.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.052+0000 7f9629ffb640 1 -- 172.21.15.3:0/1914958611 >> v1:172.21.15.203:6812/343827626 conn(0x7f9630039540 legacy=0x7f963003ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:40.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.052+0000 7f9629ffb640 10 monclient: shutdown 2024-05-21T17:21:40.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.052+0000 7f9629ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:40.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.052+0000 7f9629ffb640 1 -- 172.21.15.3:0/1914958611 >> v1:172.21.15.3:6789/0 conn(0x7f96440a0ed0 legacy=0x7f964413cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:40.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.052+0000 7f9629ffb640 1 -- 172.21.15.3:0/1914958611 shutdown_connections 2024-05-21T17:21:40.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.052+0000 7f9629ffb640 1 -- 172.21.15.3:0/1914958611 >> 172.21.15.3:0/1914958611 conn(0x7f96440a9480 msgr2=0x7f964413bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:40.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.052+0000 7f9629ffb640 1 -- 172.21.15.3:0/1914958611 shutdown_connections 2024-05-21T17:21:40.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:40.052+0000 7f9629ffb640 1 -- 172.21.15.3:0/1914958611 wait complete. 2024-05-21T17:21:40.060 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2024-05-21T17:21:40.064 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.5 2024-05-21T17:21:40.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-05-21T17:21:40.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-21T17:21:40.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-21T17:21:41.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-21T17:21:41.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-21T17:21:41.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-21T17:21:41.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient: get_monmap_and_config 2024-05-21T17:21:41.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient: monmap: 2024-05-21T17:21:41.072 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.072 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.072 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:41.072237+0000 2024-05-21T17:21:41.072 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:41.072237+0000 2024-05-21T17:21:41.072 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.073 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-21T17:21:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 1 Processor -- start 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 1 -- start start 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient: init 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient: _add_conns ranks=[7,8,4,5,0,3,2,6,1] 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient(hunting): picked mon.noname-i con 0x7f08440a4a00 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient(hunting): picked mon.noname-e con 0x7f08440a3530 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient(hunting): picked mon.noname-c con 0x7f08440a3910 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0844076a10 con 0x7f08440a3910 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f084409b7a0 con 0x7f08440a4a00 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f084409bd60 con 0x7f08440a3530 2024-05-21T17:21:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient(hunting): authenticate will time out at 968.230087s 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0842ffd640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f08440a3530 0x7f084413dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41704/0 (socket says 172.21.15.3:41704) 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0842ffd640 1 -- 172.21.15.3:0/1681357933 learned_addr learned my addr 172.21.15.3:0/1681357933 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141591234 0 0) 0x7f0844076a10 con 0x7f08440a3910 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient(hunting): my global_id is 4324 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f082c003530 con 0x7f08440a3910 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2238076223 0 0) 0x7f084409bd60 con 0x7f08440a3530 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient(hunting): my global_id is 4328 2024-05-21T17:21:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0844076a10 con 0x7f08440a3530 2024-05-21T17:21:41.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3131536053 0 0) 0x7f082c003530 con 0x7f08440a3910 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 >> v1:172.21.15.3:6793/0 conn(0x7f08440a3530 legacy=0x7f084413dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 >> v1:172.21.15.203:6792/0 conn(0x7f08440a4a00 legacy=0x7f08440a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 monclient: found mon.noname-c 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0844073f90 con 0x7f08440a3910 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f083c001bc0 con 0x7f08440a3910 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: dump: 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.083 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 5 monclient: authenticate success, global_id 4324 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f083c001de0 con 0x7f08440a3910 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 1 -- 172.21.15.3:0/1681357933 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f083c002050 con 0x7f08440a3910 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:41.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f08427fc640 10 monclient: dump: 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.086 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.068+0000 7f0849e65640 4 set_mon_vals no callback set 2024-05-21T17:21:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient: shutdown 2024-05-21T17:21:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- 172.21.15.3:0/1681357933 >> v1:172.21.15.3:6791/0 conn(0x7f08440a3910 legacy=0x7f0844141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- 172.21.15.3:0/1681357933 shutdown_connections 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- 172.21.15.3:0/1681357933 >> 172.21.15.3:0/1681357933 conn(0x7f08440a3f10 msgr2=0x7f08440a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- 172.21.15.3:0/1681357933 shutdown_connections 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- 172.21.15.3:0/1681357933 wait complete. 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient: monmap: 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 Processor -- start 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- start start 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient: init 2024-05-21T17:21:41.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient: _add_conns ranks=[3,4,0,2,8,7,6,5,1] 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient(hunting): picked mon.noname-g con 0x7f08440a3530 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient(hunting): picked mon.noname-c con 0x7f08440a3910 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient(hunting): picked mon.noname-a con 0x7f08440a4a00 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f084408f750 con 0x7f08440a4a00 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08440746c0 con 0x7f08440a3530 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0844074200 con 0x7f08440a3910 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0849e65640 10 monclient(hunting): authenticate will time out at 968.234872s 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0842ffd640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f08440a3910 0x7f08441d2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47560/0 (socket says 172.21.15.3:47560) 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f0842ffd640 1 -- 172.21.15.3:0/2482508614 learned_addr learned my addr 172.21.15.3:0/2482508614 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1837715225 0 0) 0x7f0844074200 con 0x7f08440a3910 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): my global_id is 4333 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0830003530 con 0x7f08440a3910 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4004081275 0 0) 0x7f084408f750 con 0x7f08440a4a00 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): my global_id is 4338 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0844074200 con 0x7f08440a4a00 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2796023987 0 0) 0x7f08440746c0 con 0x7f08440a3530 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 10 monclient(hunting): my global_id is 4323 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f084408f750 con 0x7f08440a3530 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3539908201 0 0) 0x7f0830003530 con 0x7f08440a3910 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08440746c0 con 0x7f08440a3910 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1250755825 0 0) 0x7f0844074200 con 0x7f08440a4a00 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.072+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0830003530 con 0x7f08440a4a00 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0838001aa0 con 0x7f08440a3910 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient(hunting): dump: 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:41.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f083c001f90 con 0x7f08440a4a00 2024-05-21T17:21:41.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient(hunting): dump: 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1595403151 0 0) 0x7f08440746c0 con 0x7f08440a3910 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 >> v1:172.21.15.203:6790/0 conn(0x7f08440a3530 legacy=0x7f08441cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 >> v1:172.21.15.3:6789/0 conn(0x7f08440a4a00 legacy=0x7f08441d64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 monclient: found mon.c 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084408f470 con 0x7f08440a3910 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 5 monclient: authenticate success, global_id 4333 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 10 monclient: _renew_subs 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08441415d0 con 0x7f08440a3910 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 10 monclient: _renew_subs 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0844140280 con 0x7f08440a3910 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0838001d20 con 0x7f08440a3910 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0841ffb640 4 set_mon_vals no callback set 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0838004de0 con 0x7f08440a3910 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084406bf50 con 0x7f08440a3910 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 10 monclient: dump: 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.100 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f083800ff60 con 0x7f08440a3910 2024-05-21T17:21:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f0838045b70 con 0x7f08440a3910 2024-05-21T17:21:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.076+0000 7f08437fe640 10 monclient: get_auth_request con 0x7f08300394a0 auth_method 0 2024-05-21T17:21:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.084+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f0838015c50 con 0x7f08440a3910 2024-05-21T17:21:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.084+0000 7f082bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.084+0000 7f082bfff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient: get_monmap_and_config 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient: monmap: 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:41.106211+0000 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:41.106211+0000 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.107 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 1 Processor -- start 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 1 -- start start 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient: init 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient: _add_conns ranks=[7,8,6,4,3,2,0,5,1] 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient(hunting): picked mon.noname-i con 0x7f7d080a5bd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient(hunting): picked mon.noname-e con 0x7f7d080a66c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient(hunting): picked mon.noname-d con 0x7f7d080a70c0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d0809e480 con 0x7f7d080a70c0 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d0809cce0 con 0x7f7d080a5bd0 2024-05-21T17:21:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d0809cfc0 con 0x7f7d080a66c0 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d0c840640 10 monclient(hunting): authenticate will time out at 968.263844s 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d05d9b640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f7d080a66c0 0x7f7d0813dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41714/0 (socket says 172.21.15.3:41714) 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.100+0000 7f7d05d9b640 1 -- 172.21.15.3:0/91249846 learned_addr learned my addr 172.21.15.3:0/91249846 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3510815768 0 0) 0x7f7d0809cfc0 con 0x7f7d080a66c0 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): my global_id is 4337 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cf0003530 con 0x7f7d080a66c0 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223862929 0 0) 0x7f7d0809e480 con 0x7f7d080a70c0 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): my global_id is 4281 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d0809cfc0 con 0x7f7d080a70c0 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 576471275 0 0) 0x7f7d0809cce0 con 0x7f7d080a5bd0 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient(hunting): my global_id is 4381 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d0809e480 con 0x7f7d080a5bd0 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1187998193 0 0) 0x7f7d0809cfc0 con 0x7f7d080a70c0 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 >> v1:172.21.15.3:6793/0 conn(0x7f7d080a66c0 legacy=0x7f7d0813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 >> v1:172.21.15.203:6792/0 conn(0x7f7d080a5bd0 legacy=0x7f7d080a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 monclient: found mon.noname-d 2024-05-21T17:21:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d080742e0 con 0x7f7d080a70c0 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7d00001ba0 con 0x7f7d080a70c0 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: dump: 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 5 monclient: authenticate success, global_id 4281 2024-05-21T17:21:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d00001d40 con 0x7f7d080a70c0 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 4 set_mon_vals no callback set 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 1 -- 172.21.15.3:0/91249846 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7d00003ce0 con 0x7f7d080a70c0 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0559a640 10 monclient: dump: 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 10 monclient: shutdown 2024-05-21T17:21:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 1 -- 172.21.15.3:0/91249846 >> v1:172.21.15.3:6792/0 conn(0x7f7d080a70c0 legacy=0x7f7d08141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 1 -- 172.21.15.3:0/91249846 shutdown_connections 2024-05-21T17:21:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 1 -- 172.21.15.3:0/91249846 >> 172.21.15.3:0/91249846 conn(0x7f7d080ab7d0 msgr2=0x7f7d080a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 1 -- 172.21.15.3:0/91249846 shutdown_connections 2024-05-21T17:21:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 1 -- 172.21.15.3:0/91249846 wait complete. 2024-05-21T17:21:41.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 10 monclient: monmap: 2024-05-21T17:21:41.120 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.120 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:41.120 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.121 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 1 Processor -- start 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 1 -- start start 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 10 monclient: init 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.104+0000 7f7d0c840640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient: _add_conns ranks=[7,8,4,2,1,3,6,0,5] 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient(hunting): picked mon.noname-i con 0x7f7d080a5bd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient(hunting): picked mon.noname-e con 0x7f7d080a66c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient(hunting): picked mon.noname-c con 0x7f7d080a70c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d080a79f0 con 0x7f7d080a70c0 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d05d9b640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f7d080a66c0 0x7f7d081d2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41722/0 (socket says 172.21.15.3:41722) 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d05d9b640 1 -- 172.21.15.3:0/2233454949 learned_addr learned my addr 172.21.15.3:0/2233454949 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d08122a90 con 0x7f7d080a5bd0 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d08142860 con 0x7f7d080a66c0 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient(hunting): authenticate will time out at 968.268590s 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3165711017 0 0) 0x7f7d08142860 con 0x7f7d080a66c0 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): my global_id is 4346 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ce8003530 con 0x7f7d080a66c0 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946871187 0 0) 0x7f7d080742e0 con 0x7f7d080a70c0 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): my global_id is 4342 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d08142860 con 0x7f7d080a70c0 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163519616 0 0) 0x7f7d08122a90 con 0x7f7d080a5bd0 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): my global_id is 4390 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d080742e0 con 0x7f7d080a5bd0 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3783991670 0 0) 0x7f7ce8003530 con 0x7f7d080a66c0 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d08122a90 con 0x7f7d080a66c0 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7cfc001660 con 0x7f7d080a66c0 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4170535045 0 0) 0x7f7d080742e0 con 0x7f7d080a5bd0 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ce8003530 con 0x7f7d080a5bd0 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7cf8001aa0 con 0x7f7d080a5bd0 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.127 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 613520430 0 0) 0x7f7d08122a90 con 0x7f7d080a66c0 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 >> v1:172.21.15.203:6792/0 conn(0x7f7d080a5bd0 legacy=0x7f7d0813b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 >> v1:172.21.15.3:6791/0 conn(0x7f7d080a70c0 legacy=0x7f7d081d5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 monclient: found mon.e 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d081419c0 con 0x7f7d080a66c0 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 5 monclient: authenticate success, global_id 4346 2024-05-21T17:21:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient: _renew_subs 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d081227b0 con 0x7f7d080a66c0 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7cfc004aa0 con 0x7f7d080a66c0 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7cfc004f90 con 0x7f7d080a66c0 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 10 monclient: dump: 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d04d99640 4 set_mon_vals no callback set 2024-05-21T17:21:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient: _renew_subs 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d08142520 con 0x7f7d080a66c0 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f7cfc010110 con 0x7f7d080a66c0 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f7cfc045b60 con 0x7f7d080a66c0 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.108+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d0806bf50 con 0x7f7d080a66c0 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.112+0000 7f7d0659c640 10 monclient: get_auth_request con 0x7f7ce80391b0 auth_method 0 2024-05-21T17:21:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.120+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f7cfc015c80 con 0x7f7d080a66c0 2024-05-21T17:21:41.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.120+0000 7f7cf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.120+0000 7f7cf6ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:41.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 10 monclient: get_monmap_and_config 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 10 monclient: monmap: 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:41.194909+0000 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:41.194909+0000 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.195 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.196 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.196 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.196 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.196 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.196 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 1 Processor -- start 2024-05-21T17:21:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 1 -- start start 2024-05-21T17:21:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 10 monclient: init 2024-05-21T17:21:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 10 monclient: _add_conns ranks=[2,1,7,5,4,3,8,6,0] 2024-05-21T17:21:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 10 monclient(hunting): picked mon.noname-b con 0x7f4208135bf0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.188+0000 7f420d160640 10 monclient(hunting): picked mon.noname-f con 0x7f4208135fd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 10 monclient(hunting): picked mon.noname-i con 0x7f420813fc90 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42080a3b80 con 0x7f4208135fd0 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4208068f90 con 0x7f4208135bf0 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42080a4650 con 0x7f420813fc90 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 10 monclient(hunting): authenticate will time out at 968.352320s 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4206d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f4208135bf0 0x7f42080a2d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47186/0 (socket says 172.21.15.3:47186) 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4206d9d640 1 -- 172.21.15.3:0/79568222 learned_addr learned my addr 172.21.15.3:0/79568222 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2297252592 0 0) 0x7f4208068f90 con 0x7f4208135bf0 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): my global_id is 4304 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41f0003530 con 0x7f4208135bf0 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962369657 0 0) 0x7f42080a4650 con 0x7f420813fc90 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): my global_id is 4399 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4208068f90 con 0x7f420813fc90 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 633801601 0 0) 0x7f42080a3b80 con 0x7f4208135fd0 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient(hunting): my global_id is 4312 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42080a4650 con 0x7f4208135fd0 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2816215850 0 0) 0x7f41f0003530 con 0x7f4208135bf0 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 >> v1:172.21.15.203:6792/0 conn(0x7f420813fc90 legacy=0x7f4208142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 >> v1:172.21.15.203:6789/0 conn(0x7f4208135fd0 legacy=0x7f42080a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 monclient: found mon.noname-b 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420809ce70 con 0x7f4208135bf0 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f41f8001840 con 0x7f4208135bf0 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: dump: 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 5 monclient: authenticate success, global_id 4304 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41f8001a60 con 0x7f4208135bf0 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 1 -- 172.21.15.3:0/79568222 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f41f8001cd0 con 0x7f4208135bf0 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f4205d9b640 10 monclient: dump: 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 4 set_mon_vals no callback set 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 10 monclient: shutdown 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 -- 172.21.15.3:0/79568222 >> v1:172.21.15.3:6790/0 conn(0x7f4208135bf0 legacy=0x7f42080a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 -- 172.21.15.3:0/79568222 shutdown_connections 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 -- 172.21.15.3:0/79568222 >> 172.21.15.3:0/79568222 conn(0x7f420806c8d0 msgr2=0x7f420806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 -- 172.21.15.3:0/79568222 shutdown_connections 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 -- 172.21.15.3:0/79568222 wait complete. 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 10 monclient: monmap: 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.208 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.208 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.208 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.192+0000 7f420d160640 1 Processor -- start 2024-05-21T17:21:41.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 1 -- start start 2024-05-21T17:21:41.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient: init 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient: _add_conns ranks=[5,0,1,6,7,4,8,2,3] 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient(hunting): picked mon.noname-h con 0x7f4208135fd0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient(hunting): picked mon.noname-a con 0x7f420813fc90 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient(hunting): picked mon.noname-f con 0x7f42081d4060 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4208136bb0 con 0x7f420813fc90 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42080a0100 con 0x7f42081d4060 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f420809e510 con 0x7f4208135fd0 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient(hunting): authenticate will time out at 968.356677s 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f420813fc90 0x7f420809f9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34954/0 (socket says 172.21.15.3:34954) 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420659c640 1 -- 172.21.15.3:0/2716976154 learned_addr learned my addr 172.21.15.3:0/2716976154 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3195776527 0 0) 0x7f4208136bb0 con 0x7f420813fc90 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): my global_id is 4347 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41f4003530 con 0x7f420813fc90 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2061543813 0 0) 0x7f41f4003530 con 0x7f420813fc90 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4208136bb0 con 0x7f420813fc90 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f41fc001b30 con 0x7f420813fc90 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.212 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219485826 0 0) 0x7f42080a0100 con 0x7f42081d4060 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): my global_id is 4321 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41f4003530 con 0x7f42081d4060 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1316523823 0 0) 0x7f420809e510 con 0x7f4208135fd0 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient(hunting): my global_id is 4433 2024-05-21T17:21:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42080a0100 con 0x7f4208135fd0 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3142979055 0 0) 0x7f4208136bb0 con 0x7f420813fc90 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 >> v1:172.21.15.203:6791/0 conn(0x7f4208135fd0 legacy=0x7f420809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 >> v1:172.21.15.203:6789/0 conn(0x7f42081d4060 legacy=0x7f42081d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 monclient: found mon.a 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42080a4440 con 0x7f420813fc90 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 5 monclient: authenticate success, global_id 4347 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient: _renew_subs 2024-05-21T17:21:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42080a0700 con 0x7f420813fc90 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient: _renew_subs 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420d160640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42080a15a0 con 0x7f420813fc90 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41fc0038d0 con 0x7f420813fc90 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f41fc003e40 con 0x7f420813fc90 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 10 monclient: dump: 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.215 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f420559a640 4 set_mon_vals no callback set 2024-05-21T17:21:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ed7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f41fc0040a0 con 0x7f420813fc90 2024-05-21T17:21:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f41fc045be0 con 0x7f420813fc90 2024-05-21T17:21:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ed7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f41ed7fa640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e0005090 con 0x7f420813fc90 2024-05-21T17:21:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.196+0000 7f4206d9d640 10 monclient: get_auth_request con 0x7f41f4038f60 auth_method 0 2024-05-21T17:21:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.208+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f41fc015dc0 con 0x7f420813fc90 2024-05-21T17:21:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.208+0000 7f41ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.208+0000 7f41ef7fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:41.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient: get_monmap_and_config 2024-05-21T17:21:41.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient: monmap: 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:41.216509+0000 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:41.216509+0000 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 1 Processor -- start 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 1 -- start start 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient: init 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient: _add_conns ranks=[2,5,0,1,6,7,3,8,4] 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce80a5bd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient(hunting): picked mon.noname-h con 0x7f4ce80a66c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce80a70c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce77fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f4ce80a5bd0 0x7f4ce80a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47200/0 (socket says 172.21.15.3:47200) 2024-05-21T17:21:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce809e480 con 0x7f4ce80a70c0 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce77fe640 1 -- 172.21.15.3:0/3629207564 learned_addr learned my addr 172.21.15.3:0/3629207564 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 1 -- 172.21.15.3:0/3629207564 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce809cce0 con 0x7f4ce80a5bd0 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 1 -- 172.21.15.3:0/3629207564 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce809cfc0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 10 monclient(hunting): authenticate will time out at 968.374152s 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3479435506 0 0) 0x7f4ce809e480 con 0x7f4ce80a70c0 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): my global_id is 4356 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cd0003530 con 0x7f4ce80a70c0 2024-05-21T17:21:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 499251077 0 0) 0x7f4ce809cce0 con 0x7f4ce80a5bd0 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): my global_id is 4313 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ce809e480 con 0x7f4ce80a5bd0 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1827904534 0 0) 0x7f4ce809cfc0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient(hunting): my global_id is 4442 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ce809cce0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2223568 0 0) 0x7f4cd0003530 con 0x7f4ce80a70c0 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 >> v1:172.21.15.203:6791/0 conn(0x7f4ce80a66c0 legacy=0x7f4ce813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 >> v1:172.21.15.3:6790/0 conn(0x7f4ce80a5bd0 legacy=0x7f4ce80a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 monclient: found mon.noname-a 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce80742e0 con 0x7f4ce80a70c0 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4ce0001b40 con 0x7f4ce80a70c0 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 5 monclient: authenticate success, global_id 4356 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4cede0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.212+0000 7f4ce67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4ce67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4ce67fc640 10 monclient: dump: 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ce0001da0 con 0x7f4ce80a70c0 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4ce67fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4ce67fc640 1 -- 172.21.15.3:0/3629207564 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4ce0003ce0 con 0x7f4ce80a70c0 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 4 set_mon_vals no callback set 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 10 monclient: shutdown 2024-05-21T17:21:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4ce67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4ce67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4ce67fc640 10 monclient: dump: 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 1 -- 172.21.15.3:0/3629207564 >> v1:172.21.15.3:6789/0 conn(0x7f4ce80a70c0 legacy=0x7f4ce8141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 1 -- 172.21.15.3:0/3629207564 shutdown_connections 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 1 -- 172.21.15.3:0/3629207564 >> 172.21.15.3:0/3629207564 conn(0x7f4ce80ab7d0 msgr2=0x7f4ce80a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 1 -- 172.21.15.3:0/3629207564 shutdown_connections 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 1 -- 172.21.15.3:0/3629207564 wait complete. 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 10 monclient: monmap: 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 1 Processor -- start 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 1 -- start start 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.216+0000 7f4cede0e640 10 monclient: init 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient: _add_conns ranks=[1,2,7,5,8,0,6,3,4] 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient(hunting): picked mon.noname-f con 0x7f4ce80a5bd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce80a66c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient(hunting): picked mon.noname-i con 0x7f4ce80a70c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ce6ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f4ce80a66c0 0x7f4ce81c9db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47206/0 (socket says 172.21.15.3:47206) 2024-05-21T17:21:41.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ce6ffd640 1 -- 172.21.15.3:0/2952343005 learned_addr learned my addr 172.21.15.3:0/2952343005 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce80a79f0 con 0x7f4ce80a5bd0 2024-05-21T17:21:41.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce813c820 con 0x7f4ce80a66c0 2024-05-21T17:21:41.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce81231d0 con 0x7f4ce80a70c0 2024-05-21T17:21:41.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient(hunting): authenticate will time out at 968.381279s 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2667845538 0 0) 0x7f4ce813c820 con 0x7f4ce80a66c0 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): my global_id is 4322 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cd4003530 con 0x7f4ce80a66c0 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479085332 0 0) 0x7f4ce80a79f0 con 0x7f4ce80a5bd0 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): my global_id is 4330 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ce813c820 con 0x7f4ce80a5bd0 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217923608 0 0) 0x7f4ce81231d0 con 0x7f4ce80a70c0 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): my global_id is 4408 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ce80a79f0 con 0x7f4ce80a70c0 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1496970206 0 0) 0x7f4cd4003530 con 0x7f4ce80a66c0 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ce81231d0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4cdc001ae0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): dump: 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3102448553 0 0) 0x7f4ce81231d0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 >> v1:172.21.15.203:6792/0 conn(0x7f4ce80a70c0 legacy=0x7f4ce81d56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 >> v1:172.21.15.203:6789/0 conn(0x7f4ce80a5bd0 legacy=0x7f4ce813c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 monclient: found mon.b 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce81234b0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 5 monclient: authenticate success, global_id 4322 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4cdc0038d0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient: _renew_subs 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce8123790 con 0x7f4ce80a66c0 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4cdc003e80 con 0x7f4ce80a66c0 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 10 monclient: dump: 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ce5ffb640 4 set_mon_vals no callback set 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient: _renew_subs 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4cede0e640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ce81ca4c0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f4cdc004020 con 0x7f4ce80a66c0 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.220+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f4cdc045bc0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.224+0000 7f4ccdffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.224+0000 7f4ccdffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.224+0000 7f4ccdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.224+0000 7f4ccdffb640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc0005090 con 0x7f4ce80a66c0 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f4cdc015bf0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7f4ce77fe640 10 monclient: get_auth_request con 0x7f4cd4038fb0 auth_method 0 2024-05-21T17:21:41.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7f4ccffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7f4ccffff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:41.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient: get_monmap_and_config 2024-05-21T17:21:41.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient: monmap: 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:41.238092+0000 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:41.238092+0000 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 1 Processor -- start 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 1 -- start start 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient: init 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient: _add_conns ranks=[6,7,3,4,2,0,8,5,1] 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient(hunting): picked mon.noname-d con 0x7fbe940a5bd0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient(hunting): picked mon.noname-i con 0x7fbe940a66c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient(hunting): picked mon.noname-g con 0x7fbe940a70c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9409e480 con 0x7fbe940a70c0 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9409cce0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe9259c640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7fbe940a5bd0 0x7fbe940a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:46024/0 (socket says 172.21.15.3:46024) 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe9259c640 1 -- 172.21.15.3:0/3995046901 learned_addr learned my addr 172.21.15.3:0/3995046901 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 1 -- 172.21.15.3:0/3995046901 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9409cfc0 con 0x7fbe940a66c0 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe98e57640 10 monclient(hunting): authenticate will time out at 968.395561s 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3334289534 0 0) 0x7fbe9409cce0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe91d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe91d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe91d9b640 10 monclient(hunting): my global_id is 4290 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.232+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe80003530 con 0x7fbe940a5bd0 2024-05-21T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3996148106 0 0) 0x7fbe9409e480 con 0x7fbe940a70c0 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient(hunting): my global_id is 4332 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe9409cce0 con 0x7fbe940a70c0 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2161195181 0 0) 0x7fbe9409cfc0 con 0x7fbe940a66c0 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient(hunting): my global_id is 4417 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe9409e480 con 0x7fbe940a66c0 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1746520295 0 0) 0x7fbe80003530 con 0x7fbe940a5bd0 2024-05-21T17:21:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 >> v1:172.21.15.203:6792/0 conn(0x7fbe940a66c0 legacy=0x7fbe9413dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 >> v1:172.21.15.203:6790/0 conn(0x7fbe940a70c0 legacy=0x7fbe94141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 monclient: found mon.noname-d 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe940742e0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbe84001840 con 0x7fbe940a5bd0 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: dump: 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 5 monclient: authenticate success, global_id 4290 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbe84001aa0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 1 -- 172.21.15.3:0/3995046901 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbe84004bf0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe91d9b640 10 monclient: dump: 2024-05-21T17:21:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.258 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.258 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:41.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 4 set_mon_vals no callback set 2024-05-21T17:21:41.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient: get_monmap_and_config 2024-05-21T17:21:41.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient: monmap: 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:41.241143+0000 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:41.241143+0000 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.259 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.260 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.260 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.260 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 10 monclient: shutdown 2024-05-21T17:21:41.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 1 -- 172.21.15.3:0/3995046901 >> v1:172.21.15.3:6792/0 conn(0x7fbe940a5bd0 legacy=0x7fbe940a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 1 -- 172.21.15.3:0/3995046901 shutdown_connections 2024-05-21T17:21:41.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 1 Processor -- start 2024-05-21T17:21:41.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 1 -- 172.21.15.3:0/3995046901 >> 172.21.15.3:0/3995046901 conn(0x7fbe940ab7d0 msgr2=0x7fbe940a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 1 -- 172.21.15.3:0/3995046901 shutdown_connections 2024-05-21T17:21:41.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 1 -- 172.21.15.3:0/3995046901 wait complete. 2024-05-21T17:21:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 1 -- start start 2024-05-21T17:21:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient: init 2024-05-21T17:21:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient: _add_conns ranks=[4,8,0,3,2,1,5,7,6] 2024-05-21T17:21:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient(hunting): picked mon.noname-c con 0x7f105c0a2ce0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient(hunting): picked mon.noname-e con 0x7f105c0a4470 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient(hunting): picked mon.noname-a con 0x7f105c09e040 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c09d380 con 0x7f105c09e040 2024-05-21T17:21:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c09d660 con 0x7f105c0a2ce0 2024-05-21T17:21:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c068530 con 0x7f105c0a4470 2024-05-21T17:21:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 10 monclient(hunting): authenticate will time out at 968.398247s 2024-05-21T17:21:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105ad9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f105c09e040 0x7f105c145810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34962/0 (socket says 172.21.15.3:34962) 2024-05-21T17:21:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105ad9d640 1 -- 172.21.15.3:0/1753401925 learned_addr learned my addr 172.21.15.3:0/1753401925 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 10 monclient: monmap: 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.264 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.265 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.265 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.265 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 854001904 0 0) 0x7f105c09d660 con 0x7f105c0a2ce0 2024-05-21T17:21:41.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): my global_id is 4351 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1044003530 con 0x7f105c0a2ce0 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3211277759 0 0) 0x7f105c09d380 con 0x7f105c09e040 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): my global_id is 4365 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f105c09d660 con 0x7f105c09e040 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2332038766 0 0) 0x7f105c068530 con 0x7f105c0a4470 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient(hunting): my global_id is 4355 2024-05-21T17:21:41.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f105c09d380 con 0x7f105c0a4470 2024-05-21T17:21:41.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 1 Processor -- start 2024-05-21T17:21:41.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 1 -- start start 2024-05-21T17:21:41.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7fbe98e57640 10 monclient: init 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1586272125 0 0) 0x7f105c09d660 con 0x7f105c09e040 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 >> v1:172.21.15.3:6793/0 conn(0x7f105c0a4470 legacy=0x7f105c1420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 >> v1:172.21.15.3:6791/0 conn(0x7f105c0a2ce0 legacy=0x7f105c0a30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 monclient: found mon.noname-a 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c0a3ec0 con 0x7f105c09e040 2024-05-21T17:21:41.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1054001ba0 con 0x7f105c09e040 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f105959a640 10 monclient: dump: 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.269 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 5 monclient: authenticate success, global_id 4365 2024-05-21T17:21:41.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.236+0000 7f1060cb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:41.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 10 monclient: _add_conns ranks=[3,1,4,2,8,0,5,6,7] 2024-05-21T17:21:41.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1054001d40 con 0x7f105c09e040 2024-05-21T17:21:41.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f105959a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f105959a640 1 -- 172.21.15.3:0/1753401925 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1054004f70 con 0x7f105c09e040 2024-05-21T17:21:41.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f105959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f105959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f105959a640 10 monclient: dump: 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.272 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.273 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.273 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.273 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.273 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.273 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 10 monclient(hunting): picked mon.noname-g con 0x7fbe940a5bd0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:41.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 10 monclient(hunting): picked mon.noname-f con 0x7fbe940a66c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:41.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 10 monclient(hunting): picked mon.noname-c con 0x7fbe940a70c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:41.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe92d9d640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fbe940a70c0 0x7fbe941d5b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47590/0 (socket says 172.21.15.3:47590) 2024-05-21T17:21:41.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe92d9d640 1 -- 172.21.15.3:0/3323355609 learned_addr learned my addr 172.21.15.3:0/3323355609 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe940a79f0 con 0x7fbe940a66c0 2024-05-21T17:21:41.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe94141b20 con 0x7fbe940a5bd0 2024-05-21T17:21:41.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe94141e00 con 0x7fbe940a70c0 2024-05-21T17:21:41.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7fbe98e57640 10 monclient(hunting): authenticate will time out at 968.401421s 2024-05-21T17:21:41.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f1060cb5640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:41.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f1060cb5640 4 set_mon_vals no callback set 2024-05-21T17:21:41.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f1060cb5640 10 monclient: shutdown 2024-05-21T17:21:41.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f1060cb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.240+0000 7f1060cb5640 1 -- 172.21.15.3:0/1753401925 >> v1:172.21.15.3:6789/0 conn(0x7f105c09e040 legacy=0x7f105c145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554634272 0 0) 0x7fbe94141b20 con 0x7fbe940a5bd0 2024-05-21T17:21:41.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): my global_id is 4341 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe80003530 con 0x7fbe940a5bd0 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3451478982 0 0) 0x7fbe94141e00 con 0x7fbe940a70c0 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): my global_id is 4360 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe94141b20 con 0x7fbe940a70c0 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4201187329 0 0) 0x7fbe9409e480 con 0x7fbe940a66c0 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): my global_id is 4339 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe94141e00 con 0x7fbe940a66c0 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3853521994 0 0) 0x7fbe80003530 con 0x7fbe940a5bd0 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe9409e480 con 0x7fbe940a5bd0 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbe84001780 con 0x7fbe940a5bd0 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): dump: 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2322671739 0 0) 0x7fbe94141e00 con 0x7fbe940a66c0 2024-05-21T17:21:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe80003530 con 0x7fbe940a66c0 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbe7c0019e0 con 0x7fbe940a66c0 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient(hunting): dump: 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 790702913 0 0) 0x7fbe9409e480 con 0x7fbe940a5bd0 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 >> v1:172.21.15.3:6791/0 conn(0x7fbe940a70c0 legacy=0x7fbe941d5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 >> v1:172.21.15.203:6789/0 conn(0x7fbe940a66c0 legacy=0x7fbe9413e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 monclient: found mon.g 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9409e6a0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 5 monclient: authenticate success, global_id 4341 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 10 monclient: _renew_subs 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9413ed80 con 0x7fbe940a5bd0 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbe84004c50 con 0x7fbe940a5bd0 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbe84001c50 con 0x7fbe940a5bd0 2024-05-21T17:21:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe8affd640 10 monclient: dump: 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 10 monclient: _renew_subs 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbe941403b0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.244+0000 7fbe98e57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.248+0000 7fbe98e57640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.248+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fbe84001960 con 0x7fbe940a5bd0 2024-05-21T17:21:41.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.248+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9406bf50 con 0x7fbe940a5bd0 2024-05-21T17:21:41.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.248+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7fbe840459e0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.248+0000 7fbe8bfff640 10 monclient: get_auth_request con 0x7fbe80039700 auth_method 0 2024-05-21T17:21:41.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7fbe90d99640 4 set_mon_vals no callback set 2024-05-21T17:21:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- 172.21.15.3:0/1753401925 shutdown_connections 2024-05-21T17:21:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- 172.21.15.3:0/1753401925 >> 172.21.15.3:0/1753401925 conn(0x7f105c06c8d0 msgr2=0x7f105c06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- 172.21.15.3:0/1753401925 shutdown_connections 2024-05-21T17:21:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- 172.21.15.3:0/1753401925 wait complete. 2024-05-21T17:21:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: monmap: 2024-05-21T17:21:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.291 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 Processor -- start 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- start start 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: init 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: _add_conns ranks=[2,7,3,4,5,1,8,6,0] 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient(hunting): picked mon.noname-b con 0x7f105c09e040 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient(hunting): picked mon.noname-i con 0x7f105c0a2ce0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient(hunting): picked mon.noname-g con 0x7f105c0a4470 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c145f50 con 0x7f105c09e040 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c146b10 con 0x7f105c0a4470 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c127830 con 0x7f105c0a2ce0 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient(hunting): authenticate will time out at 968.413659s 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f105a59c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f105c09e040 0x7f105c144370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47222/0 (socket says 172.21.15.3:47222) 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f105a59c640 1 -- 172.21.15.3:0/451402180 learned_addr learned my addr 172.21.15.3:0/451402180 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277317972 0 0) 0x7f105c145f50 con 0x7f105c09e040 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): my global_id is 4331 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f103c003530 con 0x7f105c09e040 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 600774839 0 0) 0x7f103c003530 con 0x7f105c09e040 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f105c145f50 con 0x7f105c09e040 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f104c001b40 con 0x7f105c09e040 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): dump: 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2509939250 0 0) 0x7f105c127830 con 0x7f105c0a2ce0 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient(hunting): my global_id is 4426 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f103c003530 con 0x7f105c0a2ce0 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2059820220 0 0) 0x7f105c145f50 con 0x7f105c09e040 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 >> v1:172.21.15.203:6792/0 conn(0x7f105c0a2ce0 legacy=0x7f105c144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 >> v1:172.21.15.203:6790/0 conn(0x7f105c0a4470 legacy=0x7f105c145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 monclient: found mon.b 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c1274c0 con 0x7f105c09e040 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 5 monclient: authenticate success, global_id 4331 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: _renew_subs 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f105c140cd0 con 0x7f105c09e040 2024-05-21T17:21:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: _renew_subs 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f105c1db1d0 con 0x7f105c09e040 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f104c0038d0 con 0x7f105c09e040 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1058d99640 4 set_mon_vals no callback set 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f104c003e40 con 0x7f105c09e040 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f104affd640 10 monclient: dump: 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.252+0000 7f1060cb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.256+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f104c0040a0 con 0x7f105c09e040 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.256+0000 7f1060cb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.256+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1040005090 con 0x7f105c09e040 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.256+0000 7f1059d9b640 10 monclient: get_auth_request con 0x7f103c038d10 auth_method 0 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.256+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f104c045ca0 con 0x7f105c09e040 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.260+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f104c015e80 con 0x7f105c09e040 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.260+0000 7f104affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.260+0000 7f104affd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.260+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7fbe84015d00 con 0x7fbe940a5bd0 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.260+0000 7fbe8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.260+0000 7fbe8affd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:41.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-21T17:21:41.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-21T17:21:41.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:41.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7d0806cb90 con 0x7f7d080a66c0 2024-05-21T17:21:41.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7cf6ffd640 1 -- 172.21.15.3:0/2233454949 <== mon.8 v1:172.21.15.3:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 555833778) 0x7f7cfc015580 con 0x7f7d080a66c0 2024-05-21T17:21:41.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7cf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-21T17:21:41.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7cf6ffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:41.333 INFO:teuthology.orchestra.run.smithi003.stdout:47244640258 2024-05-21T17:21:41.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 >> v1:172.21.15.203:6812/343827626 conn(0x7f7ce80391b0 legacy=0x7f7ce803b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 10 monclient: shutdown 2024-05-21T17:21:41.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 >> v1:172.21.15.3:6793/0 conn(0x7f7d080a66c0 legacy=0x7f7d081d2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 shutdown_connections 2024-05-21T17:21:41.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.328+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 >> 172.21.15.3:0/2233454949 conn(0x7f7d080ab7d0 msgr2=0x7f7d08075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.332+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 shutdown_connections 2024-05-21T17:21:41.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.332+0000 7f7d0c840640 1 -- 172.21.15.3:0/2233454949 wait complete. 2024-05-21T17:21:41.351 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.3 2024-05-21T17:21:41.351 DEBUG:teuthology.parallel:result is None 2024-05-21T17:21:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.392+0000 7f0849e65640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-21T17:21:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.392+0000 7f0849e65640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-21T17:21:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.392+0000 7f0849e65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.392+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f084413c120 con 0x7f08440a3910 2024-05-21T17:21:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.392+0000 7f082bfff640 1 -- 172.21.15.3:0/2482508614 <== mon.4 v1:172.21.15.3:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 847495621) 0x7f0838019900 con 0x7f08440a3910 2024-05-21T17:21:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.392+0000 7f082bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-21T17:21:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.392+0000 7f082bfff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:41.399 INFO:teuthology.orchestra.run.smithi003.stdout:47244640259 2024-05-21T17:21:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.396+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 >> v1:172.21.15.203:6812/343827626 conn(0x7f08300394a0 legacy=0x7f083003b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.396+0000 7f0849e65640 10 monclient: shutdown 2024-05-21T17:21:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.396+0000 7f0849e65640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.396+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 >> v1:172.21.15.3:6791/0 conn(0x7f08440a3910 legacy=0x7f08441d2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.396+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 shutdown_connections 2024-05-21T17:21:41.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.396+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 >> 172.21.15.3:0/2482508614 conn(0x7f08440a3f10 msgr2=0x7f0844141fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.396+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 shutdown_connections 2024-05-21T17:21:41.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.396+0000 7f0849e65640 1 -- 172.21.15.3:0/2482508614 wait complete. 2024-05-21T17:21:41.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-05-21T17:21:41.416 DEBUG:teuthology.parallel:result is None 2024-05-21T17:21:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.424+0000 7f1060cb5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-21T17:21:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.424+0000 7f1060cb5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-21T17:21:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.424+0000 7f1060cb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.424+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1040005380 con 0x7f105c09e040 2024-05-21T17:21:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f104affd640 1 -- 172.21.15.3:0/451402180 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 1444633209) 0x7f104c015780 con 0x7f105c09e040 2024-05-21T17:21:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f104affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-21T17:21:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f104affd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:41.432 INFO:teuthology.orchestra.run.smithi003.stdout:47244640260 2024-05-21T17:21:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 >> v1:172.21.15.203:6812/343827626 conn(0x7f103c038d10 legacy=0x7f103c03b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f1060cb5640 10 monclient: shutdown 2024-05-21T17:21:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f1060cb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 >> v1:172.21.15.3:6790/0 conn(0x7f105c09e040 legacy=0x7f105c144370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 shutdown_connections 2024-05-21T17:21:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 >> 172.21.15.3:0/451402180 conn(0x7f105c06c8d0 msgr2=0x7f105c0a1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 shutdown_connections 2024-05-21T17:21:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.428+0000 7f1060cb5640 1 -- 172.21.15.3:0/451402180 wait complete. 2024-05-21T17:21:41.448 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2024-05-21T17:21:41.448 DEBUG:teuthology.parallel:result is None 2024-05-21T17:21:41.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.468+0000 7f4ccdffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-21T17:21:41.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.468+0000 7f4ccdffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-21T17:21:41.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.468+0000 7f4ccdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:41.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.468+0000 7f4ccdffb640 1 -- 172.21.15.3:0/2952343005 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f4cc0005380 con 0x7f4ce80a66c0 2024-05-21T17:21:41.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.468+0000 7f4ccffff640 1 -- 172.21.15.3:0/2952343005 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 847495621) 0x7f4cdc0198a0 con 0x7f4ce80a66c0 2024-05-21T17:21:41.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.468+0000 7f4ccffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-21T17:21:41.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.468+0000 7f4ccffff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:41.476 INFO:teuthology.orchestra.run.smithi003.stdout:47244640259 2024-05-21T17:21:41.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.472+0000 7f4ccdffb640 1 -- 172.21.15.3:0/2952343005 >> v1:172.21.15.203:6812/343827626 conn(0x7f4cd4038fb0 legacy=0x7f4cd403b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.472+0000 7f4ccdffb640 10 monclient: shutdown 2024-05-21T17:21:41.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.472+0000 7f4ccdffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.472+0000 7f4ccdffb640 1 -- 172.21.15.3:0/2952343005 >> v1:172.21.15.3:6790/0 conn(0x7f4ce80a66c0 legacy=0x7f4ce81c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.472+0000 7f4ccdffb640 1 -- 172.21.15.3:0/2952343005 shutdown_connections 2024-05-21T17:21:41.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.472+0000 7f4ccdffb640 1 -- 172.21.15.3:0/2952343005 >> 172.21.15.3:0/2952343005 conn(0x7f4ce80ab7d0 msgr2=0x7f4ce80a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.472+0000 7f4ccdffb640 1 -- 172.21.15.3:0/2952343005 shutdown_connections 2024-05-21T17:21:41.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.472+0000 7f4ccdffb640 1 -- 172.21.15.3:0/2952343005 wait complete. 2024-05-21T17:21:41.491 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2024-05-21T17:21:41.491 DEBUG:teuthology.parallel:result is None 2024-05-21T17:21:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-21T17:21:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-21T17:21:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 1 -- 172.21.15.3:0/2716976154 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f41e0005380 con 0x7f420813fc90 2024-05-21T17:21:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ef7fe640 1 -- 172.21.15.3:0/2716976154 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1444633209) 0x7f41fc0156c0 con 0x7f420813fc90 2024-05-21T17:21:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-21T17:21:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ef7fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:41.529 INFO:teuthology.orchestra.run.smithi003.stdout:47244640260 2024-05-21T17:21:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 1 -- 172.21.15.3:0/2716976154 >> v1:172.21.15.203:6812/343827626 conn(0x7f41f4038f60 legacy=0x7f41f403b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 10 monclient: shutdown 2024-05-21T17:21:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 1 -- 172.21.15.3:0/2716976154 >> v1:172.21.15.3:6789/0 conn(0x7f420813fc90 legacy=0x7f420809f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 1 -- 172.21.15.3:0/2716976154 shutdown_connections 2024-05-21T17:21:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 1 -- 172.21.15.3:0/2716976154 >> 172.21.15.3:0/2716976154 conn(0x7f420806c8d0 msgr2=0x7f4208131160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 1 -- 172.21.15.3:0/2716976154 shutdown_connections 2024-05-21T17:21:41.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.524+0000 7f41ed7fa640 1 -- 172.21.15.3:0/2716976154 wait complete. 2024-05-21T17:21:41.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-21T17:21:41.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-21T17:21:41.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.203:6790/0 2024-05-21T17:21:41.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 --> v1:172.21.15.203:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fbe9413f3f0 con 0x7fbe940a5bd0 2024-05-21T17:21:41.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe8affd640 1 -- 172.21.15.3:0/3323355609 <== mon.3 v1:172.21.15.203:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 847495621) 0x7fbe84015600 con 0x7fbe940a5bd0 2024-05-21T17:21:41.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-21T17:21:41.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe8affd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:41.537 INFO:teuthology.orchestra.run.smithi003.stdout:47244640259 2024-05-21T17:21:41.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 >> v1:172.21.15.203:6812/343827626 conn(0x7fbe80039700 legacy=0x7fbe8003bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 10 monclient: shutdown 2024-05-21T17:21:41.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 >> v1:172.21.15.203:6790/0 conn(0x7fbe940a5bd0 legacy=0x7fbe9413df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 shutdown_connections 2024-05-21T17:21:41.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 >> 172.21.15.3:0/3323355609 conn(0x7fbe940ab7d0 msgr2=0x7fbe940a9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 shutdown_connections 2024-05-21T17:21:41.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.532+0000 7fbe98e57640 1 -- 172.21.15.3:0/3323355609 wait complete. 2024-05-21T17:21:41.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-05-21T17:21:41.547 DEBUG:teuthology.parallel:result is None 2024-05-21T17:21:41.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2024-05-21T17:21:41.554 DEBUG:teuthology.parallel:result is None 2024-05-21T17:21:41.555 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-21T17:21:41.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:21:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.696+0000 7f22a7d57640 10 monclient: get_monmap_and_config 2024-05-21T17:21:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.696+0000 7f22a7d57640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.696+0000 7f22a7d57640 10 monclient: monmap: 2024-05-21T17:21:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:41.702689+0000 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:41.702689+0000 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.704 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.696+0000 7f22a7d57640 1 Processor -- start 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.696+0000 7f22a7d57640 1 -- start start 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.696+0000 7f22a7d57640 10 monclient: init 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: _add_conns ranks=[7,3,8,1,4,2,5,0,6] 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): picked mon.noname-i con 0x7f22a00a2e70 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): picked mon.noname-g con 0x7f22a00a3960 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): picked mon.noname-e con 0x7f22a009dba0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a009ca30 con 0x7f22a00a3960 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a00a42b0 con 0x7f22a00a2e70 2024-05-21T17:21:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a009d780 con 0x7f22a009dba0 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): authenticate will time out at 968.860192s 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a62f4640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f22a009dba0 0x7f22a0141580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41734/0 (socket says 172.21.15.3:41734) 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a62f4640 1 -- 172.21.15.3:0/3760291769 learned_addr learned my addr 172.21.15.3:0/3760291769 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458941411 0 0) 0x7f22a009d780 con 0x7f22a009dba0 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient(hunting): my global_id is 4364 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f228c003530 con 0x7f22a009dba0 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3547006590 0 0) 0x7f228c003530 con 0x7f22a009dba0 2024-05-21T17:21:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 >> v1:172.21.15.203:6792/0 conn(0x7f22a00a2e70 legacy=0x7f22a00a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 >> v1:172.21.15.203:6790/0 conn(0x7f22a00a3960 legacy=0x7f22a013de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 monclient: found mon.noname-e 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a009dfb0 con 0x7f22a009dba0 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2298001bc0 con 0x7f22a009dba0 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: dump: 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 5 monclient: authenticate success, global_id 4364 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2298001de0 con 0x7f22a009dba0 2024-05-21T17:21:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 1 -- 172.21.15.3:0/3760291769 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f22980020d0 con 0x7f22a009dba0 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a4af1640 10 monclient: dump: 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 4 set_mon_vals no callback set 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: shutdown 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- 172.21.15.3:0/3760291769 >> v1:172.21.15.3:6793/0 conn(0x7f22a009dba0 legacy=0x7f22a0141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- 172.21.15.3:0/3760291769 shutdown_connections 2024-05-21T17:21:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- 172.21.15.3:0/3760291769 >> 172.21.15.3:0/3760291769 conn(0x7f22a006caa0 msgr2=0x7f22a006ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- 172.21.15.3:0/3760291769 shutdown_connections 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- 172.21.15.3:0/3760291769 wait complete. 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: monmap: 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 Processor -- start 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- start start 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: init 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: _add_conns ranks=[7,3,4,8,0,6,1,5,2] 2024-05-21T17:21:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): picked mon.noname-i con 0x7f22a009dba0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): picked mon.noname-g con 0x7f22a00a2e70 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): picked mon.noname-c con 0x7f22a00a3960 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a009e580 con 0x7f22a00a2e70 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a01cd500 con 0x7f22a00a3960 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a013fb50 con 0x7f22a009dba0 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient(hunting): authenticate will time out at 968.862883s 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a62f4640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f22a00a3960 0x7f22a01d5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47602/0 (socket says 172.21.15.3:47602) 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a62f4640 1 -- 172.21.15.3:0/1660883554 learned_addr learned my addr 172.21.15.3:0/1660883554 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54199549 0 0) 0x7f22a01cd500 con 0x7f22a00a3960 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): my global_id is 4369 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2284003560 con 0x7f22a00a3960 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 746955336 0 0) 0x7f2284003560 con 0x7f22a00a3960 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22a01cd500 con 0x7f22a00a3960 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2298001b80 con 0x7f22a00a3960 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): dump: 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899394958 0 0) 0x7f22a009e580 con 0x7f22a00a2e70 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): my global_id is 4359 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2284003560 con 0x7f22a00a2e70 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1563000608 0 0) 0x7f22a013fb50 con 0x7f22a009dba0 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient(hunting): my global_id is 4444 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22a009e580 con 0x7f22a009dba0 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 726945650 0 0) 0x7f22a01cd500 con 0x7f22a00a3960 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 >> v1:172.21.15.203:6792/0 conn(0x7f22a009dba0 legacy=0x7f22a01cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 >> v1:172.21.15.203:6790/0 conn(0x7f22a00a2e70 legacy=0x7f22a01d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 monclient: found mon.c 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a0141e40 con 0x7f22a00a3960 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 5 monclient: authenticate success, global_id 4369 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: _renew_subs 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22a01ce060 con 0x7f22a00a3960 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: _renew_subs 2024-05-21T17:21:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f22a01425c0 con 0x7f22a00a3960 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2298001fd0 con 0x7f22a00a3960 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2298001de0 con 0x7f22a00a3960 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f2297fff640 4 set_mon_vals no callback set 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.700+0000 7f22967fc640 10 monclient: dump: 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.704+0000 7f22a7d57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.704+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f2298010660 con 0x7f22a00a3960 2024-05-21T17:21:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.704+0000 7f22a7d57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.704+0000 7f22a7d57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.704+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2288005090 con 0x7f22a00a3960 2024-05-21T17:21:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.704+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f2298045f60 con 0x7f22a00a3960 2024-05-21T17:21:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.704+0000 7f22a5af3640 10 monclient: get_auth_request con 0x7f2284038f10 auth_method 0 2024-05-21T17:21:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.708+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f22980160e0 con 0x7f22a00a3960 2024-05-21T17:21:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.708+0000 7f22967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.708+0000 7f22967fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:41.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.860+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2288000ba0 con 0x7f2284038f10 2024-05-21T17:21:41.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.864+0000 7f22967fc640 1 -- 172.21.15.3:0/1660883554 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33873 (unknown 2965378022 0 110236888) 0x7f2288000ba0 con 0x7f2284038f10 2024-05-21T17:21:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:41.870 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:21:41.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.868+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 >> v1:172.21.15.203:6812/343827626 conn(0x7f2284038f10 legacy=0x7f228403b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.868+0000 7f22a7d57640 10 monclient: shutdown 2024-05-21T17:21:41.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.868+0000 7f22a7d57640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:41.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.868+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 >> v1:172.21.15.3:6791/0 conn(0x7f22a00a3960 legacy=0x7f22a01d5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:41.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.868+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 shutdown_connections 2024-05-21T17:21:41.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.868+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 >> 172.21.15.3:0/1660883554 conn(0x7f22a006caa0 msgr2=0x7f22a013be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:41.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.868+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 shutdown_connections 2024-05-21T17:21:41.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.868+0000 7f22a7d57640 1 -- 172.21.15.3:0/1660883554 wait complete. 2024-05-21T17:21:41.887 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-05-21T17:21:41.060039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":162984,"kb_used_data":808,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566068056,"statfs":{"total":579820584960,"available":579653689344,"internally_reserved":0,"allocated":827392,"data_stored":1132052,"data_compressed":17852,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-21T17:21:38.230012+0000","last_change":"2024-05-21T17:21:38.226793+0000","last_active":"2024-05-21T17:21:38.230012+0000","last_peered":"2024-05-21T17:21:38.230012+0000","last_clean":"2024-05-21T17:21:38.230012+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:21:38.230012+0000","last_undegraded":"2024-05-21T17:21:38.230012+0000","last_fullsized":"2024-05-21T17:21:38.230012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T04:06:12.903933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.250203+0000","last_change":"2024-05-21T17:21:38.830568+0000","last_active":"2024-05-21T17:21:39.250203+0000","last_peered":"2024-05-21T17:21:39.250203+0000","last_clean":"2024-05-21T17:21:39.250203+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:21:39.250203+0000","last_undegraded":"2024-05-21T17:21:39.250203+0000","last_fullsized":"2024-05-21T17:21:39.250203+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T02:04:57.235930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249638+0000","last_change":"2024-05-21T17:21:38.216293+0000","last_active":"2024-05-21T17:21:39.249638+0000","last_peered":"2024-05-21T17:21:39.249638+0000","last_clean":"2024-05-21T17:21:39.249638+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:21:39.249638+0000","last_undegraded":"2024-05-21T17:21:39.249638+0000","last_fullsized":"2024-05-21T17:21:39.249638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:53:31.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.246200+0000","last_change":"2024-05-21T17:21:38.219369+0000","last_active":"2024-05-21T17:21:39.246200+0000","last_peered":"2024-05-21T17:21:39.246200+0000","last_clean":"2024-05-21T17:21:39.246200+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:21:39.246200+0000","last_undegraded":"2024-05-21T17:21:39.246200+0000","last_fullsized":"2024-05-21T17:21:39.246200+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:24:49.428951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'72","reported_seq":102,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249849+0000","last_change":"2024-05-21T17:21:38.221127+0000","last_active":"2024-05-21T17:21:39.249849+0000","last_peered":"2024-05-21T17:21:39.249849+0000","last_clean":"2024-05-21T17:21:39.249849+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:21:39.249849+0000","last_undegraded":"2024-05-21T17:21:39.249849+0000","last_fullsized":"2024-05-21T17:21:39.249849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:12:45.986667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.246610+0000","last_change":"2024-05-21T17:21:38.219803+0000","last_active":"2024-05-21T17:21:39.246610+0000","last_peered":"2024-05-21T17:21:39.246610+0000","last_clean":"2024-05-21T17:21:39.246610+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:21:39.246610+0000","last_undegraded":"2024-05-21T17:21:39.246610+0000","last_fullsized":"2024-05-21T17:21:39.246610+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:52:49.624765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249823+0000","last_change":"2024-05-21T17:21:38.829468+0000","last_active":"2024-05-21T17:21:39.249823+0000","last_peered":"2024-05-21T17:21:39.249823+0000","last_clean":"2024-05-21T17:21:39.249823+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:21:39.249823+0000","last_undegraded":"2024-05-21T17:21:39.249823+0000","last_fullsized":"2024-05-21T17:21:39.249823+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:07:14.487479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-21T17:21:38.831733+0000","last_change":"2024-05-21T17:21:38.830884+0000","last_active":"2024-05-21T17:21:38.831733+0000","last_peered":"2024-05-21T17:21:38.831733+0000","last_clean":"2024-05-21T17:21:38.831733+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:21:38.831733+0000","last_undegraded":"2024-05-21T17:21:38.831733+0000","last_fullsized":"2024-05-21T17:21:38.831733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T01:42:21.660783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.250144+0000","last_change":"2024-05-21T17:21:38.829272+0000","last_active":"2024-05-21T17:21:39.250144+0000","last_peered":"2024-05-21T17:21:39.250144+0000","last_clean":"2024-05-21T17:21:39.250144+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:21:39.250144+0000","last_undegraded":"2024-05-21T17:21:39.250144+0000","last_fullsized":"2024-05-21T17:21:39.250144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:09:03.800931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640258,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":501075,"data_compressed":8926,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":501075,"data_compressed":8926,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":29141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:21:41.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:21:41.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.992+0000 7f2e4c87c640 10 monclient: get_monmap_and_config 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.992+0000 7f2e4c87c640 10 monclient: build_initial_monmap 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.992+0000 7f2e4c87c640 10 monclient: monmap: 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:41.998814+0000 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:41.998814+0000 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:41.999 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:42.000 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.992+0000 7f2e4c87c640 1 Processor -- start 2024-05-21T17:21:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.992+0000 7f2e4c87c640 1 -- start start 2024-05-21T17:21:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.992+0000 7f2e4c87c640 10 monclient: init 2024-05-21T17:21:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: _add_conns ranks=[8,3,5,2,4,0,7,1,6] 2024-05-21T17:21:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): picked mon.noname-e con 0x7f2e480a48c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): picked mon.noname-g con 0x7f2e480a4ca0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): picked mon.noname-h con 0x7f2e4813f210 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e480a88a0 con 0x7f2e480a4ca0 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e480a6550 con 0x7f2e4813f210 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e480a28c0 con 0x7f2e480a48c0 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): authenticate will time out at 969.156572s 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4659c640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f2e480a48c0 0x7f2e480a5e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41744/0 (socket says 172.21.15.3:41744) 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4659c640 1 -- 172.21.15.3:0/1163570981 learned_addr learned my addr 172.21.15.3:0/1163570981 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 988407566 0 0) 0x7f2e480a28c0 con 0x7f2e480a48c0 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient(hunting): my global_id is 4373 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e30003530 con 0x7f2e480a48c0 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3005287203 0 0) 0x7f2e30003530 con 0x7f2e480a48c0 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 >> v1:172.21.15.203:6791/0 conn(0x7f2e4813f210 legacy=0x7f2e48141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 >> v1:172.21.15.203:6790/0 conn(0x7f2e480a4ca0 legacy=0x7f2e4813daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: _finish_hunting 0 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 monclient: found mon.noname-e 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e480a2ba0 con 0x7f2e480a48c0 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: _finish_auth 0 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: _check_auth_tickets 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2e38001840 con 0x7f2e480a48c0 2024-05-21T17:21:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: dump: 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 5 monclient: authenticate success, global_id 4373 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2e38001a60 con 0x7f2e480a48c0 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 1 -- 172.21.15.3:0/1163570981 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2e38004bf0 con 0x7f2e480a48c0 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 4 set_mon_vals no callback set 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4559a640 10 monclient: dump: 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: shutdown 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- 172.21.15.3:0/1163570981 >> v1:172.21.15.3:6793/0 conn(0x7f2e480a48c0 legacy=0x7f2e480a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- 172.21.15.3:0/1163570981 shutdown_connections 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- 172.21.15.3:0/1163570981 >> 172.21.15.3:0/1163570981 conn(0x7f2e480ab5b0 msgr2=0x7f2e480a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- 172.21.15.3:0/1163570981 shutdown_connections 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- 172.21.15.3:0/1163570981 wait complete. 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: build_initial_monmap 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: monmap: 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 Processor -- start 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- start start 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: init 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient: _add_conns ranks=[4,6,7,2,5,0,8,3,1] 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): picked mon.noname-c con 0x7f2e480a48c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): picked mon.noname-d con 0x7f2e480a4ca0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): picked mon.noname-i con 0x7f2e4813f210 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e4809dc30 con 0x7f2e480a48c0 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e4809d810 con 0x7f2e480a4ca0 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e4813dea0 con 0x7f2e4813f210 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4c87c640 10 monclient(hunting): authenticate will time out at 969.159572s 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4659c640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f2e480a48c0 0x7f2e481cdff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47610/0 (socket says 172.21.15.3:47610) 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e4659c640 1 -- 172.21.15.3:0/2517767162 learned_addr learned my addr 172.21.15.3:0/2517767162 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 680283963 0 0) 0x7f2e4809d810 con 0x7f2e480a4ca0 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 10 monclient(hunting): my global_id is 4299 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e28003530 con 0x7f2e480a4ca0 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776528485 0 0) 0x7f2e480a2ba0 con 0x7f2e480a48c0 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:41.996+0000 7f2e36ffd640 10 monclient(hunting): my global_id is 4378 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e4809d810 con 0x7f2e480a48c0 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 792529099 0 0) 0x7f2e28003530 con 0x7f2e480a4ca0 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e480a2ba0 con 0x7f2e480a4ca0 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2e3c001a70 con 0x7f2e480a4ca0 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 455637465 0 0) 0x7f2e4809d810 con 0x7f2e480a48c0 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e28003530 con 0x7f2e480a48c0 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465798368 0 0) 0x7f2e4813dea0 con 0x7f2e4813f210 2024-05-21T17:21:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): my global_id is 4453 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e4809d810 con 0x7f2e4813f210 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2e38004b30 con 0x7f2e480a48c0 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2361325232 0 0) 0x7f2e480a2ba0 con 0x7f2e480a4ca0 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 >> v1:172.21.15.203:6792/0 conn(0x7f2e4813f210 legacy=0x7f2e481d6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 >> v1:172.21.15.3:6791/0 conn(0x7f2e480a48c0 legacy=0x7f2e481cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 monclient: found mon.d 2024-05-21T17:21:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e480a7110 con 0x7f2e480a4ca0 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 5 monclient: authenticate success, global_id 4299 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 10 monclient: _renew_subs 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2e480a5190 con 0x7f2e480a4ca0 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 10 monclient: _renew_subs 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2e4813d7a0 con 0x7f2e480a4ca0 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2e3c001d10 con 0x7f2e480a4ca0 2024-05-21T17:21:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2e3c003c30 con 0x7f2e480a4ca0 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e44d99640 4 set_mon_vals no callback set 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 10 monclient: dump: 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.3:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e2c005090 con 0x7f2e480a4ca0 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f2e3c003e50 con 0x7f2e480a4ca0 2024-05-21T17:21:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f2e3c044a40 con 0x7f2e480a4ca0 2024-05-21T17:21:42.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.000+0000 7f2e4659c640 10 monclient: get_auth_request con 0x7f2e280393f0 auth_method 0 2024-05-21T17:21:42.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.004+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mon.6 v1:172.21.15.3:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f2e3c014be0 con 0x7f2e480a4ca0 2024-05-21T17:21:42.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.004+0000 7f2e36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.004+0000 7f2e36ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:42.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.156+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2e2c000ba0 con 0x7f2e280393f0 2024-05-21T17:21:42.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e36ffd640 1 -- 172.21.15.3:0/2517767162 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33873 (unknown 2965378022 0 110236888) 0x7f2e2c000ba0 con 0x7f2e280393f0 2024-05-21T17:21:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:42.166 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:21:42.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 >> v1:172.21.15.203:6812/343827626 conn(0x7f2e280393f0 legacy=0x7f2e2803b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e4c87c640 10 monclient: shutdown 2024-05-21T17:21:42.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e4c87c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:42.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 >> v1:172.21.15.3:6792/0 conn(0x7f2e480a4ca0 legacy=0x7f2e481d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 shutdown_connections 2024-05-21T17:21:42.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 >> 172.21.15.3:0/2517767162 conn(0x7f2e480ab5b0 msgr2=0x7f2e48132ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:42.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 shutdown_connections 2024-05-21T17:21:42.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.160+0000 7f2e4c87c640 1 -- 172.21.15.3:0/2517767162 wait complete. 2024-05-21T17:21:42.182 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-05-21T17:21:41.060039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":162984,"kb_used_data":808,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566068056,"statfs":{"total":579820584960,"available":579653689344,"internally_reserved":0,"allocated":827392,"data_stored":1132052,"data_compressed":17852,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-21T17:21:38.230012+0000","last_change":"2024-05-21T17:21:38.226793+0000","last_active":"2024-05-21T17:21:38.230012+0000","last_peered":"2024-05-21T17:21:38.230012+0000","last_clean":"2024-05-21T17:21:38.230012+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:21:38.230012+0000","last_undegraded":"2024-05-21T17:21:38.230012+0000","last_fullsized":"2024-05-21T17:21:38.230012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T04:06:12.903933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.250203+0000","last_change":"2024-05-21T17:21:38.830568+0000","last_active":"2024-05-21T17:21:39.250203+0000","last_peered":"2024-05-21T17:21:39.250203+0000","last_clean":"2024-05-21T17:21:39.250203+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:21:39.250203+0000","last_undegraded":"2024-05-21T17:21:39.250203+0000","last_fullsized":"2024-05-21T17:21:39.250203+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T02:04:57.235930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249638+0000","last_change":"2024-05-21T17:21:38.216293+0000","last_active":"2024-05-21T17:21:39.249638+0000","last_peered":"2024-05-21T17:21:39.249638+0000","last_clean":"2024-05-21T17:21:39.249638+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:21:39.249638+0000","last_undegraded":"2024-05-21T17:21:39.249638+0000","last_fullsized":"2024-05-21T17:21:39.249638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:53:31.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.246200+0000","last_change":"2024-05-21T17:21:38.219369+0000","last_active":"2024-05-21T17:21:39.246200+0000","last_peered":"2024-05-21T17:21:39.246200+0000","last_clean":"2024-05-21T17:21:39.246200+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:21:39.246200+0000","last_undegraded":"2024-05-21T17:21:39.246200+0000","last_fullsized":"2024-05-21T17:21:39.246200+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:24:49.428951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'72","reported_seq":102,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249849+0000","last_change":"2024-05-21T17:21:38.221127+0000","last_active":"2024-05-21T17:21:39.249849+0000","last_peered":"2024-05-21T17:21:39.249849+0000","last_clean":"2024-05-21T17:21:39.249849+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:21:39.249849+0000","last_undegraded":"2024-05-21T17:21:39.249849+0000","last_fullsized":"2024-05-21T17:21:39.249849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:12:45.986667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.246610+0000","last_change":"2024-05-21T17:21:38.219803+0000","last_active":"2024-05-21T17:21:39.246610+0000","last_peered":"2024-05-21T17:21:39.246610+0000","last_clean":"2024-05-21T17:21:39.246610+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:21:39.246610+0000","last_undegraded":"2024-05-21T17:21:39.246610+0000","last_fullsized":"2024-05-21T17:21:39.246610+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:52:49.624765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249823+0000","last_change":"2024-05-21T17:21:38.829468+0000","last_active":"2024-05-21T17:21:39.249823+0000","last_peered":"2024-05-21T17:21:39.249823+0000","last_clean":"2024-05-21T17:21:39.249823+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:21:39.249823+0000","last_undegraded":"2024-05-21T17:21:39.249823+0000","last_fullsized":"2024-05-21T17:21:39.249823+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:07:14.487479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-21T17:21:38.831733+0000","last_change":"2024-05-21T17:21:38.830884+0000","last_active":"2024-05-21T17:21:38.831733+0000","last_peered":"2024-05-21T17:21:38.831733+0000","last_clean":"2024-05-21T17:21:38.831733+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:21:38.831733+0000","last_undegraded":"2024-05-21T17:21:38.831733+0000","last_fullsized":"2024-05-21T17:21:38.831733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T01:42:21.660783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.250144+0000","last_change":"2024-05-21T17:21:38.829272+0000","last_active":"2024-05-21T17:21:39.250144+0000","last_peered":"2024-05-21T17:21:39.250144+0000","last_clean":"2024-05-21T17:21:39.250144+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:21:39.250144+0000","last_undegraded":"2024-05-21T17:21:39.250144+0000","last_fullsized":"2024-05-21T17:21:39.250144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:09:03.800931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640258,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":501075,"data_compressed":8926,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":501075,"data_compressed":8926,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":29141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:21:42.183 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-21T17:21:42.183 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-21T17:21:42.183 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-21T17:21:42.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-21T17:21:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient: get_monmap_and_config 2024-05-21T17:21:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient: build_initial_monmap 2024-05-21T17:21:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient: monmap: 2024-05-21T17:21:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:42.301705+0000 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:42.301705+0000 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:42.303 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 1 Processor -- start 2024-05-21T17:21:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 1 -- start start 2024-05-21T17:21:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient: init 2024-05-21T17:21:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient: _add_conns ranks=[3,1,6,5,7,4,8,0,2] 2024-05-21T17:21:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient(hunting): picked mon.noname-g con 0x7f589c1388b0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient(hunting): picked mon.noname-f con 0x7f589c1393a0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient(hunting): picked mon.noname-d con 0x7f589c09db30 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589c09e720 con 0x7f589c1393a0 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589c0a41b0 con 0x7f589c1388b0 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589c0a43f0 con 0x7f589c09db30 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f58a1d4c640 10 monclient(hunting): authenticate will time out at 969.459227s 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589bfff640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f589c09db30 0x7f589c141580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:46044/0 (socket says 172.21.15.3:46044) 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589bfff640 1 -- 172.21.15.3:0/753384440 learned_addr learned my addr 172.21.15.3:0/753384440 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827598172 0 0) 0x7f589c0a43f0 con 0x7f589c09db30 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 10 monclient(hunting): my global_id is 4308 2024-05-21T17:21:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5888003530 con 0x7f589c09db30 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3967908764 0 0) 0x7f5888003530 con 0x7f589c09db30 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 >> v1:172.21.15.203:6790/0 conn(0x7f589c1388b0 legacy=0x7f589c138c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 >> v1:172.21.15.203:6789/0 conn(0x7f589c1393a0 legacy=0x7f589c13de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 1 monclient: found mon.noname-d 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.296+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589c069140 con 0x7f589c09db30 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 5 monclient: authenticate success, global_id 4308 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f588c001bc0 con 0x7f589c09db30 2024-05-21T17:21:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: dump: 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f588c001de0 con 0x7f589c09db30 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 1 -- 172.21.15.3:0/753384440 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f588c004f70 con 0x7f589c09db30 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 4 set_mon_vals no callback set 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589a7fc640 10 monclient: dump: 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: shutdown 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- 172.21.15.3:0/753384440 >> v1:172.21.15.3:6792/0 conn(0x7f589c09db30 legacy=0x7f589c141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- 172.21.15.3:0/753384440 shutdown_connections 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- 172.21.15.3:0/753384440 >> 172.21.15.3:0/753384440 conn(0x7f589c06c8d0 msgr2=0x7f589c06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- 172.21.15.3:0/753384440 shutdown_connections 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- 172.21.15.3:0/753384440 wait complete. 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: build_initial_monmap 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: monmap: 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 Processor -- start 2024-05-21T17:21:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- start start 2024-05-21T17:21:42.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: init 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: _add_conns ranks=[2,3,0,6,4,8,5,1,7] 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient(hunting): picked mon.noname-b con 0x7f589c09db30 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient(hunting): picked mon.noname-g con 0x7f589c1388b0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient(hunting): picked mon.noname-a con 0x7f589c1393a0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589c1384d0 con 0x7f589c1393a0 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589c09e8f0 con 0x7f589c09db30 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589c137970 con 0x7f589c1388b0 2024-05-21T17:21:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient(hunting): authenticate will time out at 969.462157s 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589bfff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f589c1393a0 0x7f589c1d5be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34982/0 (socket says 172.21.15.3:34982) 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589bfff640 1 -- 172.21.15.3:0/3460692961 learned_addr learned my addr 172.21.15.3:0/3460692961 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f589b7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f589c09db30 0x7f589c136030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47224/0 (socket says 172.21.15.3:47224) 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828497002 0 0) 0x7f589c1384d0 con 0x7f589c1393a0 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): my global_id is 4374 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f587c003560 con 0x7f589c1393a0 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755464146 0 0) 0x7f589c09e8f0 con 0x7f589c09db30 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): my global_id is 4340 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f589c1384d0 con 0x7f589c09db30 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 389944198 0 0) 0x7f589c137970 con 0x7f589c1388b0 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): my global_id is 4377 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f589c09e8f0 con 0x7f589c1388b0 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1294579186 0 0) 0x7f587c003560 con 0x7f589c1393a0 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f589c137970 con 0x7f589c1393a0 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f588c004eb0 con 0x7f589c1393a0 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): dump: 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3950687300 0 0) 0x7f589c1384d0 con 0x7f589c09db30 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f587c003560 con 0x7f589c09db30 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5894001a60 con 0x7f589c09db30 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient(hunting): dump: 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4239290906 0 0) 0x7f589c137970 con 0x7f589c1393a0 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 >> v1:172.21.15.203:6790/0 conn(0x7f589c1388b0 legacy=0x7f589c136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 >> v1:172.21.15.3:6790/0 conn(0x7f589c09db30 legacy=0x7f589c136030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 monclient: found mon.a 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589c13a3b0 con 0x7f589c1393a0 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient: _finish_auth 0 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 5 monclient: authenticate success, global_id 4374 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: _renew_subs 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f589c142760 con 0x7f589c1393a0 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: _renew_subs 2024-05-21T17:21:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f589c136e50 con 0x7f589c1393a0 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f588c001520 con 0x7f589c1393a0 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f588c005230 con 0x7f589c1393a0 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f58997fa640 4 set_mon_vals no callback set 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.300+0000 7f5893fff640 10 monclient: dump: 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.304+0000 7f58a1d4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.304+0000 7f58a1d4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.304+0000 7f58a1d4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.304+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5880005090 con 0x7f589c1393a0 2024-05-21T17:21:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.304+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f588c0103b0 con 0x7f589c1393a0 2024-05-21T17:21:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.304+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f588c045ee0 con 0x7f589c1393a0 2024-05-21T17:21:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.304+0000 7f589b7fe640 10 monclient: get_auth_request con 0x7f587c039410 auth_method 0 2024-05-21T17:21:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.308+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f588c015fa0 con 0x7f589c1393a0 2024-05-21T17:21:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.308+0000 7f5893fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.308+0000 7f5893fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:42.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.504+0000 7f58a1d4c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-05-21T17:21:42.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.504+0000 7f58a1d4c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-05-21T17:21:42.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.504+0000 7f58a1d4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:42.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.504+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5880005380 con 0x7f589c1393a0 2024-05-21T17:21:42.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.504+0000 7f5893fff640 1 -- 172.21.15.3:0/3460692961 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f588c019bf0 con 0x7f589c1393a0 2024-05-21T17:21:42.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.504+0000 7f5893fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-05-21T17:21:42.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.504+0000 7f5893fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:42.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:42.511 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-21T17:21:42.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.508+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 >> v1:172.21.15.203:6812/343827626 conn(0x7f587c039410 legacy=0x7f587c03b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.508+0000 7f58a1d4c640 10 monclient: shutdown 2024-05-21T17:21:42.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.508+0000 7f58a1d4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:42.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.508+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 >> v1:172.21.15.3:6789/0 conn(0x7f589c1393a0 legacy=0x7f589c1d5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.508+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 shutdown_connections 2024-05-21T17:21:42.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.508+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 >> 172.21.15.3:0/3460692961 conn(0x7f589c06c8d0 msgr2=0x7f589c0a6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:42.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.508+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 shutdown_connections 2024-05-21T17:21:42.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.508+0000 7f58a1d4c640 1 -- 172.21.15.3:0/3460692961 wait complete. 2024-05-21T17:21:42.527 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-21T17:21:42.527 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-05-21T17:21:42.540 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-05-21T17:21:42.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-21T17:21:42.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.644+0000 7f354465c640 10 monclient: get_monmap_and_config 2024-05-21T17:21:42.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.644+0000 7f354465c640 10 monclient: build_initial_monmap 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.644+0000 7f354465c640 10 monclient: monmap: 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:42.650847+0000 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:42.650847+0000 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:42.652 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:42.652 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:42.652 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:42.652 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:42.652 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:42.652 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.644+0000 7f354465c640 1 Processor -- start 2024-05-21T17:21:42.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.644+0000 7f354465c640 1 -- start start 2024-05-21T17:21:42.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.644+0000 7f354465c640 10 monclient: init 2024-05-21T17:21:42.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:42.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient: _add_conns ranks=[3,5,1,7,2,8,4,6,0] 2024-05-21T17:21:42.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient(hunting): picked mon.noname-g con 0x7f353c0734d0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:42.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient(hunting): picked mon.noname-h con 0x7f353c0a0c20 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:42.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient(hunting): picked mon.noname-f con 0x7f353c070e70 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f353c0a20d0 con 0x7f353c070e70 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f353c0a23b0 con 0x7f353c0734d0 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f353c0a3430 con 0x7f353c0a0c20 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient(hunting): authenticate will time out at 969.808316s 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35423f8640 1 --1- >> v1:172.21.15.203:6790/0 conn(0x7f353c0734d0 0x7f353c0738b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6790/0 says I am v1:172.21.15.3:40882/0 (socket says 172.21.15.3:40882) 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35423f8640 1 -- 172.21.15.3:0/3569691135 learned_addr learned my addr 172.21.15.3:0/3569691135 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2644903384 0 0) 0x7f353c0a20d0 con 0x7f353c070e70 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): my global_id is 4357 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3524003530 con 0x7f353c070e70 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2843923432 0 0) 0x7f353c0a23b0 con 0x7f353c0734d0 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): my global_id is 4386 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f353c0a20d0 con 0x7f353c0734d0 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2734814813 0 0) 0x7f353c0a3430 con 0x7f353c0a0c20 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient(hunting): my global_id is 4460 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f353c0a23b0 con 0x7f353c0a0c20 2024-05-21T17:21:42.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2892722163 0 0) 0x7f3524003530 con 0x7f353c070e70 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 >> v1:172.21.15.203:6791/0 conn(0x7f353c0a0c20 legacy=0x7f353c13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 >> v1:172.21.15.203:6790/0 conn(0x7f353c0734d0 legacy=0x7f353c0738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: _finish_hunting 0 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 monclient: found mon.noname-f 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.203:6789/0 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353c073fc0 con 0x7f353c070e70 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: _finish_auth 0 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: _check_auth_tickets 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f352c001840 con 0x7f353c070e70 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-05-21T17:21:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: dump: 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 5 monclient: authenticate success, global_id 4357 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f352c001a60 con 0x7f353c070e70 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 4 set_mon_vals no callback set 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 1 -- 172.21.15.3:0/3569691135 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f352c003960 con 0x7f353c070e70 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f35413f6640 10 monclient: dump: 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.665 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient: shutdown 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- 172.21.15.3:0/3569691135 >> v1:172.21.15.203:6789/0 conn(0x7f353c070e70 legacy=0x7f353c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- 172.21.15.3:0/3569691135 shutdown_connections 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- 172.21.15.3:0/3569691135 >> 172.21.15.3:0/3569691135 conn(0x7f353c0a9640 msgr2=0x7f353c0a5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- 172.21.15.3:0/3569691135 shutdown_connections 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- 172.21.15.3:0/3569691135 wait complete. 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient: build_initial_monmap 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient: monmap: 2024-05-21T17:21:42.666 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 Processor -- start 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 1 -- start start 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.648+0000 7f354465c640 10 monclient: init 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient: _add_conns ranks=[4,3,0,5,8,6,1,7,2] 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient(hunting): picked mon.noname-c con 0x7f353c070e70 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient(hunting): picked mon.noname-g con 0x7f353c0734d0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient(hunting): picked mon.noname-a con 0x7f353c140380 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f353c13d8b0 con 0x7f353c140380 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f353c13caa0 con 0x7f353c0734d0 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f353c13fba0 con 0x7f353c070e70 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient(hunting): authenticate will time out at 969.812445s 2024-05-21T17:21:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f35423f8640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f353c070e70 0x7f353c13bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47626/0 (socket says 172.21.15.3:47626) 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f35423f8640 1 -- 172.21.15.3:0/1904324749 learned_addr learned my addr 172.21.15.3:0/1904324749 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3924269186 0 0) 0x7f353c13fba0 con 0x7f353c070e70 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): my global_id is 4387 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3528003530 con 0x7f353c070e70 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022637940 0 0) 0x7f353c13d8b0 con 0x7f353c140380 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): my global_id is 4383 2024-05-21T17:21:42.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f353c13fba0 con 0x7f353c140380 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1196244591 0 0) 0x7f3528003530 con 0x7f353c070e70 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f353c13d8b0 con 0x7f353c070e70 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3272075185 0 0) 0x7f353c13fba0 con 0x7f353c140380 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3528003530 con 0x7f353c140380 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3534001ea0 con 0x7f353c070e70 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): dump: 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.670 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f352c001800 con 0x7f353c140380 2024-05-21T17:21:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): dump: 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.672 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2189125077 0 0) 0x7f353c13caa0 con 0x7f353c0734d0 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient(hunting): my global_id is 4395 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f353c13fba0 con 0x7f353c0734d0 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1143379930 0 0) 0x7f353c13d8b0 con 0x7f353c070e70 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 >> v1:172.21.15.203:6790/0 conn(0x7f353c0734d0 legacy=0x7f353c13c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 >> v1:172.21.15.3:6789/0 conn(0x7f353c140380 legacy=0x7f353c1d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 monclient: found mon.c 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353c13f300 con 0x7f353c070e70 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient: _finish_auth 0 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 5 monclient: authenticate success, global_id 4387 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient: _renew_subs 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f353c137490 con 0x7f353c070e70 2024-05-21T17:21:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient: _renew_subs 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f353c1d52b0 con 0x7f353c070e70 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3534004e20 con 0x7f353c070e70 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3534002040 con 0x7f353c070e70 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f3540bf5640 4 set_mon_vals no callback set 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 10 monclient: dump: 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.675 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f351c005090 con 0x7f353c070e70 2024-05-21T17:21:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f3534010370 con 0x7f353c070e70 2024-05-21T17:21:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f3534046010 con 0x7f353c070e70 2024-05-21T17:21:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.652+0000 7f3541bf7640 10 monclient: get_auth_request con 0x7f3528039370 auth_method 0 2024-05-21T17:21:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.656+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f3534016070 con 0x7f353c070e70 2024-05-21T17:21:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.656+0000 7f353affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.656+0000 7f353affd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:42.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.812+0000 7f354465c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:42.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.812+0000 7f354465c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:42.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.812+0000 7f354465c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.812+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f351c005380 con 0x7f353c070e70 2024-05-21T17:21:42.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f353affd640 1 -- 172.21.15.3:0/1904324749 <== mon.4 v1:172.21.15.3:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10761 (unknown 4221056075 0 2217018945) 0x7f3534019d20 con 0x7f353c070e70 2024-05-21T17:21:42.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f353affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:21:42.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f353affd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:42.821 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:42.821 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":15,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","created":"2024-05-21T17:21:26.465014+0000","modified":"2024-05-21T17:21:39.218346+0000","last_up_change":"2024-05-21T17:21:35.154389+0000","last_in_change":"2024-05-21T17:21:28.263976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-21T17:21:36.581878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-21T17:21:37.102413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7dfaea0b-2159-4f6e-86a6-da6637052771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6800","nonce":857120552}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6801","nonce":857120552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6803","nonce":857120552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6802","nonce":857120552}]},"public_addr":"172.21.15.3:6800/857120552","cluster_addr":"172.21.15.3:6801/857120552","heartbeat_back_addr":"172.21.15.3:6803/857120552","heartbeat_front_addr":"172.21.15.3:6802/857120552","state":["exists","up"]},{"osd":1,"uuid":"28d053ac-034d-43ed-a83e-9ee62a426b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6808","nonce":4137544047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6809","nonce":4137544047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6811","nonce":4137544047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6810","nonce":4137544047}]},"public_addr":"172.21.15.3:6808/4137544047","cluster_addr":"172.21.15.3:6809/4137544047","heartbeat_back_addr":"172.21.15.3:6811/4137544047","heartbeat_front_addr":"172.21.15.3:6810/4137544047","state":["exists","up"]},{"osd":2,"uuid":"9de90871-f66b-4a85-ac48-772541c5b23d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6804","nonce":517111994}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6805","nonce":517111994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6807","nonce":517111994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6806","nonce":517111994}]},"public_addr":"172.21.15.3:6804/517111994","cluster_addr":"172.21.15.3:6805/517111994","heartbeat_back_addr":"172.21.15.3:6807/517111994","heartbeat_front_addr":"172.21.15.3:6806/517111994","state":["exists","up"]},{"osd":3,"uuid":"f8431d10-6318-458b-9a94-51fc47273f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":1468009857}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1468009857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1468009857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":1468009857}]},"public_addr":"172.21.15.203:6804/1468009857","cluster_addr":"172.21.15.203:6805/1468009857","heartbeat_back_addr":"172.21.15.203:6807/1468009857","heartbeat_front_addr":"172.21.15.203:6806/1468009857","state":["exists","up"]},{"osd":4,"uuid":"028748a9-3ca4-4730-960e-8913c9504a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":4020495055}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":4020495055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":4020495055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":4020495055}]},"public_addr":"172.21.15.203:6800/4020495055","cluster_addr":"172.21.15.203:6801/4020495055","heartbeat_back_addr":"172.21.15.203:6803/4020495055","heartbeat_front_addr":"172.21.15.203:6802/4020495055","state":["exists","up"]},{"osd":5,"uuid":"87412b84-97f1-4350-9034-87f8f33da44a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":2732955435}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":2732955435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":2732955435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":2732955435}]},"public_addr":"172.21.15.203:6808/2732955435","cluster_addr":"172.21.15.203:6809/2732955435","heartbeat_back_addr":"172.21.15.203:6811/2732955435","heartbeat_front_addr":"172.21.15.203:6810/2732955435","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.740995+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.776640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.734901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.745181+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.676894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.827672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T17:21:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 >> v1:172.21.15.203:6812/343827626 conn(0x7f3528039370 legacy=0x7f352803b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f354465c640 10 monclient: shutdown 2024-05-21T17:21:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f354465c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 >> v1:172.21.15.3:6791/0 conn(0x7f353c070e70 legacy=0x7f353c13bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 shutdown_connections 2024-05-21T17:21:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 >> 172.21.15.3:0/1904324749 conn(0x7f353c0a9640 msgr2=0x7f353c0a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 shutdown_connections 2024-05-21T17:21:42.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.816+0000 7f354465c640 1 -- 172.21.15.3:0/1904324749 wait complete. 2024-05-21T17:21:42.838 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-05-21T17:21:36.581878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-05-21T17:21:37.102413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-21T17:21:42.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-05-21T17:21:42.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient: get_monmap_and_config 2024-05-21T17:21:42.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient: build_initial_monmap 2024-05-21T17:21:42.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient: monmap: 2024-05-21T17:21:42.961 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:42.961791+0000 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:42.961791+0000 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:42.963 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:42.963 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:42.963 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 1 Processor -- start 2024-05-21T17:21:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 1 -- start start 2024-05-21T17:21:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient: init 2024-05-21T17:21:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient: _add_conns ranks=[2,0,3,6,8,1,4,5,7] 2024-05-21T17:21:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient(hunting): picked mon.noname-b con 0x7f6bcc0a62c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient(hunting): picked mon.noname-a con 0x7f6bcc0a3040 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient(hunting): picked mon.noname-g con 0x7f6bcc13d1f0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bcc0a2310 con 0x7f6bcc0a3040 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bcc137680 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bcc0a1620 con 0x7f6bcc13d1f0 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.956+0000 7f6bd1557640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient(hunting): authenticate will time out at 970.120013s 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bca7fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6bcc0a3040 0x7f6bcc13bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34998/0 (socket says 172.21.15.3:34998) 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bcaffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6bcc0a62c0 0x7f6bcc073c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47240/0 (socket says 172.21.15.3:47240) 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bca7fc640 1 -- 172.21.15.3:0/873793212 learned_addr learned my addr 172.21.15.3:0/873793212 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1359325302 0 0) 0x7f6bcc137680 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient(hunting): my global_id is 4349 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb4003530 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330130629 0 0) 0x7f6bc0001050 con 0x7f6bcc0a3040 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient(hunting): my global_id is 4392 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bcc137680 con 0x7f6bcc0a3040 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1106721127 0 0) 0x7f6bb4003530 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 >> v1:172.21.15.203:6790/0 conn(0x7f6bcc13d1f0 legacy=0x7f6bcc13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 >> v1:172.21.15.3:6789/0 conn(0x7f6bcc0a3040 legacy=0x7f6bcc13bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 monclient: found mon.noname-b 2024-05-21T17:21:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc0a1270 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 5 monclient: authenticate success, global_id 4349 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6bbc001bc0 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: dump: 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bbc001d60 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 1 -- 172.21.15.3:0/873793212 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6bbc004eb0 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 4 set_mon_vals no callback set 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bc9ffb640 10 monclient: dump: 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient: shutdown 2024-05-21T17:21:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- 172.21.15.3:0/873793212 >> v1:172.21.15.3:6790/0 conn(0x7f6bcc0a62c0 legacy=0x7f6bcc073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- 172.21.15.3:0/873793212 shutdown_connections 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- 172.21.15.3:0/873793212 >> 172.21.15.3:0/873793212 conn(0x7f6bcc06cd50 msgr2=0x7f6bcc0a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- 172.21.15.3:0/873793212 shutdown_connections 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- 172.21.15.3:0/873793212 wait complete. 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient: build_initial_monmap 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient: monmap: 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 Processor -- start 2024-05-21T17:21:42.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- start start 2024-05-21T17:21:42.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient: init 2024-05-21T17:21:42.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:42.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient: _add_conns ranks=[4,7,8,2,3,6,1,5,0] 2024-05-21T17:21:42.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient(hunting): picked mon.noname-c con 0x7f6bcc0a3040 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:42.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient(hunting): picked mon.noname-i con 0x7f6bcc0a62c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:42.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient(hunting): picked mon.noname-e con 0x7f6bcc13d1f0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:42.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bcc0a1840 con 0x7f6bcc0a3040 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bcc1217b0 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bcc122f40 con 0x7f6bcc13d1f0 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bd1557640 10 monclient(hunting): authenticate will time out at 970.123140s 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bcaffd640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f6bcc0a3040 0x7f6bcc135580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47634/0 (socket says 172.21.15.3:47634) 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bcb7fe640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f6bcc13d1f0 0x7f6bcc1d5e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41746/0 (socket says 172.21.15.3:41746) 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bcaffd640 1 -- 172.21.15.3:0/2471781901 learned_addr learned my addr 172.21.15.3:0/2471781901 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3083396639 0 0) 0x7f6bcc122f40 con 0x7f6bcc13d1f0 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): my global_id is 4382 2024-05-21T17:21:42.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb8003530 con 0x7f6bcc13d1f0 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546874738 0 0) 0x7f6bcc0a1270 con 0x7f6bcc0a3040 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): my global_id is 4396 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bcc122f40 con 0x7f6bcc0a3040 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1335732443 0 0) 0x7f6bcc122f40 con 0x7f6bcc0a3040 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bcc0a1270 con 0x7f6bcc0a3040 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6bbc004ef0 con 0x7f6bcc0a3040 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): dump: 2024-05-21T17:21:42.980 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.981 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3623641030 0 0) 0x7f6bb8003530 con 0x7f6bcc13d1f0 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bcc122f40 con 0x7f6bcc13d1f0 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6bc4001960 con 0x7f6bcc13d1f0 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 10 monclient(hunting): dump: 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.982 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.960+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1471891789 0 0) 0x7f6bcc1217b0 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:42.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient(hunting): my global_id is 4462 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb8003530 con 0x7f6bcc0a62c0 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2939148319 0 0) 0x7f6bcc0a1270 con 0x7f6bcc0a3040 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 >> v1:172.21.15.3:6793/0 conn(0x7f6bcc13d1f0 legacy=0x7f6bcc1d5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 >> v1:172.21.15.203:6792/0 conn(0x7f6bcc0a62c0 legacy=0x7f6bcc1d2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 monclient: found mon.c 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc121500 con 0x7f6bcc0a3040 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 5 monclient: authenticate success, global_id 4396 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 10 monclient: _renew_subs 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6bcc1cd660 con 0x7f6bcc0a3040 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 10 monclient: _renew_subs 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f6bcc135c90 con 0x7f6bcc0a3040 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bbc001b80 con 0x7f6bcc0a3040 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6bbc001d30 con 0x7f6bcc0a3040 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bc97fa640 4 set_mon_vals no callback set 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 10 monclient: dump: 2024-05-21T17:21:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f6bbc010490 con 0x7f6bcc0a3040 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bac005090 con 0x7f6bcc0a3040 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f6bbc045ea0 con 0x7f6bcc0a3040 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.964+0000 7f6bca7fc640 10 monclient: get_auth_request con 0x7f6bb8039360 auth_method 0 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.968+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f6bcc0a1840 con 0x7f6bcc0a3040 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.968+0000 7f6bb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:42.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:42.968+0000 7f6bb37fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:43.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.124+0000 7f6bd1557640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-05-21T17:21:43.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.124+0000 7f6bd1557640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-05-21T17:21:43.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.124+0000 7f6bd1557640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:43.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.124+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f6bac005b90 con 0x7f6bcc0a3040 2024-05-21T17:21:43.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.124+0000 7f6bb37fe640 1 -- 172.21.15.3:0/2471781901 <== mon.4 v1:172.21.15.3:6791/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) ==== 92+0+10 (unknown 946793192 0 265718887) 0x7f6bbc0160f0 con 0x7f6bcc0a3040 2024-05-21T17:21:43.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.124+0000 7f6bb37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-05-21T17:21:43.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.124+0000 7f6bb37fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:43.131 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2024-05-21T17:21:43.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.128+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 >> v1:172.21.15.203:6812/343827626 conn(0x7f6bb8039360 legacy=0x7f6bb803b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.128+0000 7f6bd1557640 10 monclient: shutdown 2024-05-21T17:21:43.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.128+0000 7f6bd1557640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:43.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.128+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 >> v1:172.21.15.3:6791/0 conn(0x7f6bcc0a3040 legacy=0x7f6bcc135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.128+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 shutdown_connections 2024-05-21T17:21:43.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.128+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 >> 172.21.15.3:0/2471781901 conn(0x7f6bcc06cd50 msgr2=0x7f6bcc06bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:43.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.128+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 shutdown_connections 2024-05-21T17:21:43.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.128+0000 7f6bd1557640 1 -- 172.21.15.3:0/2471781901 wait complete. 2024-05-21T17:21:43.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-21T17:21:43.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.260+0000 7fde06e19640 10 monclient: get_monmap_and_config 2024-05-21T17:21:43.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.260+0000 7fde06e19640 10 monclient: build_initial_monmap 2024-05-21T17:21:43.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.260+0000 7fde06e19640 10 monclient: monmap: 2024-05-21T17:21:43.267 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:43.267 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.267 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:43.266396+0000 2024-05-21T17:21:43.267 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:43.266396+0000 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:43.268 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.264+0000 7fde06e19640 1 Processor -- start 2024-05-21T17:21:43.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.264+0000 7fde06e19640 1 -- start start 2024-05-21T17:21:43.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.264+0000 7fde06e19640 10 monclient: init 2024-05-21T17:21:43.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:43.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: _add_conns ranks=[4,6,2,0,3,1,8,5,7] 2024-05-21T17:21:43.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): picked mon.noname-c con 0x7fde00072e80 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:43.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): picked mon.noname-d con 0x7fde000a3220 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:43.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): picked mon.noname-b con 0x7fde00073970 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:43.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde000a2560 con 0x7fde00073970 2024-05-21T17:21:43.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde000a2840 con 0x7fde00072e80 2024-05-21T17:21:43.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde00074120 con 0x7fde000a3220 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): authenticate will time out at 970.428536s 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde05e17640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fde00072e80 0x7fde00073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47638/0 (socket says 172.21.15.3:47638) 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde05e17640 1 -- 172.21.15.3:0/3461619928 learned_addr learned my addr 172.21.15.3:0/3461619928 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965147205 0 0) 0x7fddf80014a0 con 0x7fde00073970 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient(hunting): my global_id is 4358 2024-05-21T17:21:43.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddec003530 con 0x7fde00073970 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3268383998 0 0) 0x7fde00074120 con 0x7fde000a3220 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient(hunting): my global_id is 4317 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf80014a0 con 0x7fde000a3220 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 442423583 0 0) 0x7fddec003530 con 0x7fde00073970 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 >> v1:172.21.15.3:6792/0 conn(0x7fde000a3220 legacy=0x7fde0013dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 >> v1:172.21.15.3:6791/0 conn(0x7fde00072e80 legacy=0x7fde00073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: _finish_hunting 0 2024-05-21T17:21:43.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 monclient: found mon.noname-b 2024-05-21T17:21:43.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:43.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:43.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde000a4130 con 0x7fde00073970 2024-05-21T17:21:43.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: _finish_auth 0 2024-05-21T17:21:43.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: _check_auth_tickets 2024-05-21T17:21:43.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:43.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 5 monclient: authenticate success, global_id 4358 2024-05-21T17:21:43.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fddf8001d90 con 0x7fde00073970 2024-05-21T17:21:43.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:43.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:43.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: dump: 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.279 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fddf8001fb0 con 0x7fde00073970 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 4 set_mon_vals no callback set 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 1 -- 172.21.15.3:0/3461619928 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fddf8003cc0 con 0x7fde00073970 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde04e15640 10 monclient: dump: 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: shutdown 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- 172.21.15.3:0/3461619928 >> v1:172.21.15.3:6790/0 conn(0x7fde00073970 legacy=0x7fde00141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- 172.21.15.3:0/3461619928 shutdown_connections 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- 172.21.15.3:0/3461619928 >> 172.21.15.3:0/3461619928 conn(0x7fde000a7380 msgr2=0x7fde000a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- 172.21.15.3:0/3461619928 shutdown_connections 2024-05-21T17:21:43.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- 172.21.15.3:0/3461619928 wait complete. 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: build_initial_monmap 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: monmap: 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:43.286 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 Processor -- start 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- start start 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: init 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient: _add_conns ranks=[0,4,5,6,1,2,7,8,3] 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): picked mon.noname-a con 0x7fde00072e80 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): picked mon.noname-c con 0x7fde00073970 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): picked mon.noname-h con 0x7fde000a3220 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0013c2a0 con 0x7fde00072e80 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde001396b0 con 0x7fde00073970 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde000a0810 con 0x7fde000a3220 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.268+0000 7fde06e19640 10 monclient(hunting): authenticate will time out at 970.431914s 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde05616640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fde00073970 0x7fde00138fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47654/0 (socket says 172.21.15.3:47654) 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde05616640 1 -- 172.21.15.3:0/1554671789 learned_addr learned my addr 172.21.15.3:0/1554671789 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014102063 0 0) 0x7fde000a2840 con 0x7fde00072e80 2024-05-21T17:21:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): my global_id is 4401 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdde4003530 con 0x7fde00072e80 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 120474563 0 0) 0x7fde001396b0 con 0x7fde00073970 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): my global_id is 4405 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde000a2840 con 0x7fde00073970 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1787375346 0 0) 0x7fdde4003530 con 0x7fde00072e80 2024-05-21T17:21:43.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde001396b0 con 0x7fde00072e80 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fddfc001a50 con 0x7fde00072e80 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): dump: 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1963791824 0 0) 0x7fde000a2840 con 0x7fde00073970 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdde4003530 con 0x7fde00073970 2024-05-21T17:21:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fddf0001a80 con 0x7fde00073970 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient(hunting): dump: 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1238376574 0 0) 0x7fde001396b0 con 0x7fde00072e80 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 >> v1:172.21.15.203:6791/0 conn(0x7fde000a3220 legacy=0x7fde001d5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 >> v1:172.21.15.3:6791/0 conn(0x7fde00073970 legacy=0x7fde00138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 monclient: found mon.a 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde00142910 con 0x7fde00072e80 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 5 monclient: authenticate success, global_id 4401 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 10 monclient: _renew_subs 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fde00139b20 con 0x7fde00072e80 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 10 monclient: _renew_subs 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fde0013be80 con 0x7fde00072e80 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fddfc001cd0 con 0x7fde00072e80 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fddfc004ea0 con 0x7fde00072e80 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf7fff640 4 set_mon_vals no callback set 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 10 monclient: dump: 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0006bf50 con 0x7fde00072e80 2024-05-21T17:21:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fddfc0050c0 con 0x7fde00072e80 2024-05-21T17:21:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7fddfc045c80 con 0x7fde00072e80 2024-05-21T17:21:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.272+0000 7fde05616640 10 monclient: get_auth_request con 0x7fdde40392c0 auth_method 0 2024-05-21T17:21:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.276+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7fde0013c2a0 con 0x7fde00072e80 2024-05-21T17:21:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.276+0000 7fddf67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.276+0000 7fddf67fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-05-21T17:21:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-05-21T17:21:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fde0013b5e0 con 0x7fde00072e80 2024-05-21T17:21:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fddf67fc640 1 -- 172.21.15.3:0/1554671789 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) ==== 93+0+10 (unknown 554344602 0 2170607528) 0x7fddfc015da0 con 0x7fde00072e80 2024-05-21T17:21:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fddf67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-05-21T17:21:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fddf67fc640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:43.442 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-21T17:21:43.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 >> v1:172.21.15.203:6812/343827626 conn(0x7fdde40392c0 legacy=0x7fdde403b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 10 monclient: shutdown 2024-05-21T17:21:43.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 >> v1:172.21.15.3:6789/0 conn(0x7fde00072e80 legacy=0x7fde00138890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 shutdown_connections 2024-05-21T17:21:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 >> 172.21.15.3:0/1554671789 conn(0x7fde000a7380 msgr2=0x7fde000a4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 shutdown_connections 2024-05-21T17:21:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.436+0000 7fde06e19640 1 -- 172.21.15.3:0/1554671789 wait complete. 2024-05-21T17:21:43.458 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-05-21T17:21:43.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:21:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.572+0000 7f344b573640 10 monclient: get_monmap_and_config 2024-05-21T17:21:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.572+0000 7f344b573640 10 monclient: build_initial_monmap 2024-05-21T17:21:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.572+0000 7f344b573640 10 monclient: monmap: 2024-05-21T17:21:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:43.578702+0000 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:43.578702+0000 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:43.596 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.572+0000 7f344b573640 1 Processor -- start 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- start start 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: init 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: _add_conns ranks=[2,8,7,6,3,4,1,0,5] 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): picked mon.noname-b con 0x7f34440734d0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): picked mon.noname-e con 0x7f34440a0c20 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): picked mon.noname-i con 0x7f3444070e70 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34440a20d0 con 0x7f34440734d0 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34440a23b0 con 0x7f3444070e70 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34440a3430 con 0x7f34440a0c20 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:43.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): authenticate will time out at 970.736846s 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f3448b0e640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f34440a0c20 0x7f344413bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:41756/0 (socket says 172.21.15.3:41756) 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344930f640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f34440734d0 0x7f34440738b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47258/0 (socket says 172.21.15.3:47258) 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f3448b0e640 1 -- 172.21.15.3:0/3999772577 learned_addr learned my addr 172.21.15.3:0/3999772577 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 434204492 0 0) 0x7f34440a3430 con 0x7f34440a0c20 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient(hunting): my global_id is 4391 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3430003530 con 0x7f34440a0c20 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2040254888 0 0) 0x7f34440a20d0 con 0x7f34440734d0 2024-05-21T17:21:43.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient(hunting): my global_id is 4367 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34440a3430 con 0x7f34440734d0 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2187418733 0 0) 0x7f3430003530 con 0x7f34440a0c20 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 >> v1:172.21.15.203:6792/0 conn(0x7f3444070e70 legacy=0x7f344413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 >> v1:172.21.15.3:6790/0 conn(0x7f34440734d0 legacy=0x7f34440738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 monclient: found mon.noname-e 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:21:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3444073fc0 con 0x7f34440a0c20 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: _finish_auth 0 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3434001840 con 0x7f34440a0c20 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 5 monclient: authenticate success, global_id 4391 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: dump: 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.600 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f34340019e0 con 0x7f34440a0c20 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 1 -- 172.21.15.3:0/3999772577 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3434004bf0 con 0x7f34440a0c20 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 4 set_mon_vals no callback set 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f343bfff640 10 monclient: dump: 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: shutdown 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- 172.21.15.3:0/3999772577 >> v1:172.21.15.3:6793/0 conn(0x7f34440a0c20 legacy=0x7f344413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- 172.21.15.3:0/3999772577 shutdown_connections 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- 172.21.15.3:0/3999772577 >> 172.21.15.3:0/3999772577 conn(0x7f34440a9640 msgr2=0x7f34440a5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- 172.21.15.3:0/3999772577 shutdown_connections 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- 172.21.15.3:0/3999772577 wait complete. 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: build_initial_monmap 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: monmap: 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 Processor -- start 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- start start 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: init 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient: _add_conns ranks=[6,4,1,3,2,0,8,7,5] 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): picked mon.noname-d con 0x7f3444070e70 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): picked mon.noname-c con 0x7f34440734d0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): picked mon.noname-f con 0x7f34441d3be0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3444120b70 con 0x7f34441d3be0 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34441ce7b0 con 0x7f34440734d0 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34440a13e0 con 0x7f3444070e70 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344b573640 10 monclient(hunting): authenticate will time out at 970.739860s 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344930f640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f3444070e70 0x7f34441cd990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:46064/0 (socket says 172.21.15.3:46064) 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f3448b0e640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f34440734d0 0x7f34441ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:47670/0 (socket says 172.21.15.3:47670) 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.576+0000 7f344930f640 1 -- 172.21.15.3:0/1039655637 learned_addr learned my addr 172.21.15.3:0/1039655637 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549425393 0 0) 0x7f34441ce7b0 con 0x7f34440734d0 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): my global_id is 4414 2024-05-21T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3428003530 con 0x7f34440734d0 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096684064 0 0) 0x7f34440a13e0 con 0x7f3444070e70 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): my global_id is 4326 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34441ce7b0 con 0x7f3444070e70 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 980453945 0 0) 0x7f3428003530 con 0x7f34440734d0 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34440a13e0 con 0x7f34440734d0 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3434001840 con 0x7f34440734d0 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): dump: 2024-05-21T17:21:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3441276217 0 0) 0x7f34441ce7b0 con 0x7f3444070e70 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3428003530 con 0x7f3444070e70 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3440001470 con 0x7f3444070e70 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): dump: 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220051239 0 0) 0x7f3444120b70 con 0x7f34441d3be0 2024-05-21T17:21:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient(hunting): my global_id is 4366 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34441ce7b0 con 0x7f34441d3be0 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1360785020 0 0) 0x7f34440a13e0 con 0x7f34440734d0 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 >> v1:172.21.15.3:6792/0 conn(0x7f3444070e70 legacy=0x7f34441cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 >> v1:172.21.15.203:6789/0 conn(0x7f34441d3be0 legacy=0x7f34441d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 monclient: found mon.c 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3444074260 con 0x7f34440734d0 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 5 monclient: authenticate success, global_id 4414 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 10 monclient: _renew_subs 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f344406d870 con 0x7f34440734d0 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 10 monclient: _renew_subs 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f344406e120 con 0x7f34440734d0 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3434001260 con 0x7f34440734d0 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3434004de0 con 0x7f34440734d0 2024-05-21T17:21:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f343b7fe640 4 set_mon_vals no callback set 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 10 monclient: dump: 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f3434004f80 con 0x7f34440734d0 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f3434045a30 con 0x7f34440734d0 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3424005090 con 0x7f34440734d0 2024-05-21T17:21:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.580+0000 7f344930f640 10 monclient: get_auth_request con 0x7f3428039400 auth_method 0 2024-05-21T17:21:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.584+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f3434015c70 con 0x7f34440734d0 2024-05-21T17:21:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.584+0000 7f3439ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.584+0000 7f3439ffb640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:43.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.748+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3424000ba0 con 0x7f3428039400 2024-05-21T17:21:43.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.748+0000 7f3439ffb640 1 -- 172.21.15.3:0/1039655637 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33828 (unknown 2965378022 0 2668438972) 0x7f3424000ba0 con 0x7f3428039400 2024-05-21T17:21:43.756 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:43.756 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:21:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.752+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 >> v1:172.21.15.203:6812/343827626 conn(0x7f3428039400 legacy=0x7f342803b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.752+0000 7f344b573640 10 monclient: shutdown 2024-05-21T17:21:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.752+0000 7f344b573640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.752+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 >> v1:172.21.15.3:6791/0 conn(0x7f34440734d0 legacy=0x7f34441ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.752+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 shutdown_connections 2024-05-21T17:21:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.752+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 >> 172.21.15.3:0/1039655637 conn(0x7f34440a9640 msgr2=0x7f34440a4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.752+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 shutdown_connections 2024-05-21T17:21:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.752+0000 7f344b573640 1 -- 172.21.15.3:0/1039655637 wait complete. 2024-05-21T17:21:43.774 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-05-21T17:21:43.060675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163016,"kb_used_data":872,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566068024,"statfs":{"total":579820584960,"available":579653656576,"internally_reserved":0,"allocated":892928,"data_stored":1285862,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000635"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.626711+0000","last_change":"2024-05-21T17:21:38.226793+0000","last_active":"2024-05-21T17:21:39.626711+0000","last_peered":"2024-05-21T17:21:39.626711+0000","last_clean":"2024-05-21T17:21:39.626711+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:21:39.626711+0000","last_undegraded":"2024-05-21T17:21:39.626711+0000","last_fullsized":"2024-05-21T17:21:39.626711+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T21:23:30.360529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.250203+0000","last_change":"2024-05-21T17:21:38.830568+0000","last_active":"2024-05-21T17:21:39.250203+0000","last_peered":"2024-05-21T17:21:39.250203+0000","last_clean":"2024-05-21T17:21:39.250203+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:21:39.250203+0000","last_undegraded":"2024-05-21T17:21:39.250203+0000","last_fullsized":"2024-05-21T17:21:39.250203+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T02:04:57.235930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249638+0000","last_change":"2024-05-21T17:21:38.216293+0000","last_active":"2024-05-21T17:21:39.249638+0000","last_peered":"2024-05-21T17:21:39.249638+0000","last_clean":"2024-05-21T17:21:39.249638+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:21:39.249638+0000","last_undegraded":"2024-05-21T17:21:39.249638+0000","last_fullsized":"2024-05-21T17:21:39.249638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:53:31.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.246200+0000","last_change":"2024-05-21T17:21:38.219369+0000","last_active":"2024-05-21T17:21:39.246200+0000","last_peered":"2024-05-21T17:21:39.246200+0000","last_clean":"2024-05-21T17:21:39.246200+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:21:39.246200+0000","last_undegraded":"2024-05-21T17:21:39.246200+0000","last_fullsized":"2024-05-21T17:21:39.246200+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:24:49.428951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'72","reported_seq":102,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249849+0000","last_change":"2024-05-21T17:21:38.221127+0000","last_active":"2024-05-21T17:21:39.249849+0000","last_peered":"2024-05-21T17:21:39.249849+0000","last_clean":"2024-05-21T17:21:39.249849+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:21:39.249849+0000","last_undegraded":"2024-05-21T17:21:39.249849+0000","last_fullsized":"2024-05-21T17:21:39.249849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:12:45.986667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.246610+0000","last_change":"2024-05-21T17:21:38.219803+0000","last_active":"2024-05-21T17:21:39.246610+0000","last_peered":"2024-05-21T17:21:39.246610+0000","last_clean":"2024-05-21T17:21:39.246610+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:21:39.246610+0000","last_undegraded":"2024-05-21T17:21:39.246610+0000","last_fullsized":"2024-05-21T17:21:39.246610+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:52:49.624765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249823+0000","last_change":"2024-05-21T17:21:38.829468+0000","last_active":"2024-05-21T17:21:39.249823+0000","last_peered":"2024-05-21T17:21:39.249823+0000","last_clean":"2024-05-21T17:21:39.249823+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:21:39.249823+0000","last_undegraded":"2024-05-21T17:21:39.249823+0000","last_fullsized":"2024-05-21T17:21:39.249823+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:07:14.487479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.626751+0000","last_change":"2024-05-21T17:21:38.830884+0000","last_active":"2024-05-21T17:21:39.626751+0000","last_peered":"2024-05-21T17:21:39.626751+0000","last_clean":"2024-05-21T17:21:39.626751+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:21:39.626751+0000","last_undegraded":"2024-05-21T17:21:39.626751+0000","last_fullsized":"2024-05-21T17:21:39.626751+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:36:41.643111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.250144+0000","last_change":"2024-05-21T17:21:38.829272+0000","last_active":"2024-05-21T17:21:39.250144+0000","last_peered":"2024-05-21T17:21:39.250144+0000","last_clean":"2024-05-21T17:21:39.250144+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:21:39.250144+0000","last_undegraded":"2024-05-21T17:21:39.250144+0000","last_fullsized":"2024-05-21T17:21:39.250144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:09:03.800931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":566929,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":566929,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:21:43.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:21:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.892+0000 7f0ece6b1640 10 monclient: get_monmap_and_config 2024-05-21T17:21:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.892+0000 7f0ece6b1640 10 monclient: build_initial_monmap 2024-05-21T17:21:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.892+0000 7f0ece6b1640 10 monclient: monmap: 2024-05-21T17:21:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:43.899611+0000 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:43.899611+0000 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:43.901 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 Processor -- start 2024-05-21T17:21:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- start start 2024-05-21T17:21:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient: init 2024-05-21T17:21:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient: _add_conns ranks=[8,1,2,3,6,5,4,7,0] 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient(hunting): picked mon.noname-e con 0x7f0ec8135f20 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient(hunting): picked mon.noname-f con 0x7f0ec8136300 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec80a3dd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec809cbc0 con 0x7f0ec8136300 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec80a0250 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec809e480 con 0x7f0ec8135f20 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient(hunting): authenticate will time out at 971.057301s 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec7fff640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f0ec8135f20 0x7f0ec809fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:52740/0 (socket says 172.21.15.3:52740) 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec7fff640 1 -- 172.21.15.3:0/54823480 learned_addr learned my addr 172.21.15.3:0/54823480 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0eccc4e640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f0ec80a3dd0 0x7f0ec813e0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52668/0 (socket says 172.21.15.3:52668) 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3203192235 0 0) 0x7f0ec809e480 con 0x7f0ec8135f20 2024-05-21T17:21:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient(hunting): my global_id is 4400 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eb0005740 con 0x7f0ec8135f20 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1800739920 0 0) 0x7f0ec80a0250 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient(hunting): my global_id is 4376 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ec809e480 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3509291492 0 0) 0x7f0ec809e480 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 >> v1:172.21.15.3:6793/0 conn(0x7f0ec8135f20 legacy=0x7f0ec809fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 >> v1:172.21.15.203:6789/0 conn(0x7f0ec8136300 legacy=0x7f0ec813a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 monclient: found mon.noname-b 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec809ea40 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0ec0001ba0 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: dump: 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.910 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 5 monclient: authenticate success, global_id 4376 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ec0001d40 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 1 -- 172.21.15.3:0/54823480 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0ec0004b70 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ec6ffd640 10 monclient: dump: 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.911 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 4 set_mon_vals no callback set 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient: shutdown 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- 172.21.15.3:0/54823480 >> v1:172.21.15.3:6790/0 conn(0x7f0ec80a3dd0 legacy=0x7f0ec813e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- 172.21.15.3:0/54823480 shutdown_connections 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- 172.21.15.3:0/54823480 >> 172.21.15.3:0/54823480 conn(0x7f0ec806c8d0 msgr2=0x7f0ec806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- 172.21.15.3:0/54823480 shutdown_connections 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 -- 172.21.15.3:0/54823480 wait complete. 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient: build_initial_monmap 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 10 monclient: monmap: 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.896+0000 7f0ece6b1640 1 Processor -- start 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 1 -- start start 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: init 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: _add_conns ranks=[0,1,3,6,5,7,8,4,2] 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec80a3dd0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient(hunting): picked mon.noname-f con 0x7f0ec8135f20 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient(hunting): picked mon.noname-g con 0x7f0ec8136300 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec8121d20 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec809d7b0 con 0x7f0ec8135f20 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec80741c0 con 0x7f0ec8136300 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient(hunting): authenticate will time out at 971.060516s 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec7fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0ec80a3dd0 0x7f0ec81c9840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47626/0 (socket says 172.21.15.3:47626) 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec7fff640 1 -- 172.21.15.3:0/3277446577 learned_addr learned my addr 172.21.15.3:0/3277446577 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114828421 0 0) 0x7f0ec8121d20 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient(hunting): _negotiate 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient(hunting): my global_id is 4410 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eb4003530 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 364411323 0 0) 0x7f0eb4003530 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ec8121d20 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0eb8001680 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient(hunting): dump: 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.916 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1373281471 0 0) 0x7f0ec8121d20 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 >> v1:172.21.15.203:6790/0 conn(0x7f0ec8136300 legacy=0x7f0ec81d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 >> v1:172.21.15.203:6789/0 conn(0x7f0ec8135f20 legacy=0x7f0ec81ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient: _finish_hunting 0 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 monclient: found mon.a 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec81215d0 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient: _finish_auth 0 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient: _check_auth_tickets 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 5 monclient: authenticate success, global_id 4410 2024-05-21T17:21:43.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: _renew_subs 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec8123650 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: _renew_subs 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ec8134360 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0eb8004aa0 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f0eb8004f90 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec67fc640 4 set_mon_vals no callback set 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 10 monclient: dump: 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:43.920 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:43.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:43.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea8005090 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f0eb8010190 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f0eb8045e00 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.900+0000 7f0ec77fe640 10 monclient: get_auth_request con 0x7f0eb4038be0 auth_method 0 2024-05-21T17:21:43.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.904+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f0eb8015f20 con 0x7f0ec80a3dd0 2024-05-21T17:21:43.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.904+0000 7f0ec4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:43.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:43.904+0000 7f0ec4ff9640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:44.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.060+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ea8000ba0 con 0x7f0eb4038be0 2024-05-21T17:21:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.064+0000 7f0ec4ff9640 1 -- 172.21.15.3:0/3277446577 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33828 (unknown 2965378022 0 2668438972) 0x7f0ea8000ba0 con 0x7f0eb4038be0 2024-05-21T17:21:44.070 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:21:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:21:44.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.064+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 >> v1:172.21.15.203:6812/343827626 conn(0x7f0eb4038be0 legacy=0x7f0eb403b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.064+0000 7f0ece6b1640 10 monclient: shutdown 2024-05-21T17:21:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.064+0000 7f0ece6b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.064+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 >> v1:172.21.15.3:6789/0 conn(0x7f0ec80a3dd0 legacy=0x7f0ec81c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.064+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 shutdown_connections 2024-05-21T17:21:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.064+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 >> 172.21.15.3:0/3277446577 conn(0x7f0ec806c8d0 msgr2=0x7f0ec8134cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.068+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 shutdown_connections 2024-05-21T17:21:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.068+0000 7f0ece6b1640 1 -- 172.21.15.3:0/3277446577 wait complete. 2024-05-21T17:21:44.087 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-05-21T17:21:43.060675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163016,"kb_used_data":872,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566068024,"statfs":{"total":579820584960,"available":579653656576,"internally_reserved":0,"allocated":892928,"data_stored":1285862,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000635"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.626711+0000","last_change":"2024-05-21T17:21:38.226793+0000","last_active":"2024-05-21T17:21:39.626711+0000","last_peered":"2024-05-21T17:21:39.626711+0000","last_clean":"2024-05-21T17:21:39.626711+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:21:39.626711+0000","last_undegraded":"2024-05-21T17:21:39.626711+0000","last_fullsized":"2024-05-21T17:21:39.626711+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T21:23:30.360529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.250203+0000","last_change":"2024-05-21T17:21:38.830568+0000","last_active":"2024-05-21T17:21:39.250203+0000","last_peered":"2024-05-21T17:21:39.250203+0000","last_clean":"2024-05-21T17:21:39.250203+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:21:39.250203+0000","last_undegraded":"2024-05-21T17:21:39.250203+0000","last_fullsized":"2024-05-21T17:21:39.250203+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T02:04:57.235930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249638+0000","last_change":"2024-05-21T17:21:38.216293+0000","last_active":"2024-05-21T17:21:39.249638+0000","last_peered":"2024-05-21T17:21:39.249638+0000","last_clean":"2024-05-21T17:21:39.249638+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:21:39.249638+0000","last_undegraded":"2024-05-21T17:21:39.249638+0000","last_fullsized":"2024-05-21T17:21:39.249638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:53:31.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.246200+0000","last_change":"2024-05-21T17:21:38.219369+0000","last_active":"2024-05-21T17:21:39.246200+0000","last_peered":"2024-05-21T17:21:39.246200+0000","last_clean":"2024-05-21T17:21:39.246200+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:21:39.246200+0000","last_undegraded":"2024-05-21T17:21:39.246200+0000","last_fullsized":"2024-05-21T17:21:39.246200+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:24:49.428951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'72","reported_seq":102,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249849+0000","last_change":"2024-05-21T17:21:38.221127+0000","last_active":"2024-05-21T17:21:39.249849+0000","last_peered":"2024-05-21T17:21:39.249849+0000","last_clean":"2024-05-21T17:21:39.249849+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:21:39.249849+0000","last_undegraded":"2024-05-21T17:21:39.249849+0000","last_fullsized":"2024-05-21T17:21:39.249849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:12:45.986667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.246610+0000","last_change":"2024-05-21T17:21:38.219803+0000","last_active":"2024-05-21T17:21:39.246610+0000","last_peered":"2024-05-21T17:21:39.246610+0000","last_clean":"2024-05-21T17:21:39.246610+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:21:39.246610+0000","last_undegraded":"2024-05-21T17:21:39.246610+0000","last_fullsized":"2024-05-21T17:21:39.246610+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:52:49.624765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.249823+0000","last_change":"2024-05-21T17:21:38.829468+0000","last_active":"2024-05-21T17:21:39.249823+0000","last_peered":"2024-05-21T17:21:39.249823+0000","last_clean":"2024-05-21T17:21:39.249823+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:21:39.249823+0000","last_undegraded":"2024-05-21T17:21:39.249823+0000","last_fullsized":"2024-05-21T17:21:39.249823+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:07:14.487479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.626751+0000","last_change":"2024-05-21T17:21:38.830884+0000","last_active":"2024-05-21T17:21:39.626751+0000","last_peered":"2024-05-21T17:21:39.626751+0000","last_clean":"2024-05-21T17:21:39.626751+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:21:39.626751+0000","last_undegraded":"2024-05-21T17:21:39.626751+0000","last_fullsized":"2024-05-21T17:21:39.626751+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:36:41.643111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-21T17:21:39.250144+0000","last_change":"2024-05-21T17:21:38.829272+0000","last_active":"2024-05-21T17:21:39.250144+0000","last_peered":"2024-05-21T17:21:39.250144+0000","last_clean":"2024-05-21T17:21:39.250144+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:21:39.250144+0000","last_undegraded":"2024-05-21T17:21:39.250144+0000","last_fullsized":"2024-05-21T17:21:39.250144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:09:03.800931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":566929,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":566929,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:21:44.087 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-05-21T17:21:44.088 DEBUG:tasks.mon_thrash:Yielding 2024-05-21T17:21:44.088 INFO:teuthology.run_tasks:Running task exec... 2024-05-21T17:21:44.113 INFO:teuthology.task.exec:Executing custom commands... 2024-05-21T17:21:44.113 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:21:44.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20' 2024-05-21T17:21:44.114 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-05-21T17:21:44.114 INFO:tasks.mon_thrash.mon_thrasher:seed: 1716312104, revive delay: 90.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-05-21T17:21:44.114 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-05-21T17:21:44.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient: get_monmap_and_config 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient: build_initial_monmap 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient: monmap: 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:44.232944+0000 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:44.232944+0000 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:44.235 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 Processor -- start 2024-05-21T17:21:44.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- start start 2024-05-21T17:21:44.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient: init 2024-05-21T17:21:44.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:44.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient: _add_conns ranks=[2,7,8,4,3,1,5,6,0] 2024-05-21T17:21:44.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient(hunting): picked mon.noname-b con 0x7f387c0a6030 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:44.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient(hunting): picked mon.noname-i con 0x7f387c0a2b10 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:44.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient(hunting): picked mon.noname-e con 0x7f387c13f300 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f387c06cb30 con 0x7f387c0a6030 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f387c06cfb0 con 0x7f387c0a2b10 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f387c0a11d0 con 0x7f387c13f300 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient(hunting): authenticate will time out at 971.390364s 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3881a83640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f387c0a6030 0x7f387c0a2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52670/0 (socket says 172.21.15.3:52670) 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3881a83640 1 -- 172.21.15.3:0/565979352 learned_addr learned my addr 172.21.15.3:0/565979352 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190855096 0 0) 0x7f387c06cb30 con 0x7f387c0a6030 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient(hunting): my global_id is 4385 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3868003530 con 0x7f387c0a6030 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1423216797 0 0) 0x7f387c0a11d0 con 0x7f387c13f300 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient(hunting): my global_id is 4418 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f387c06cb30 con 0x7f387c13f300 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 861682329 0 0) 0x7f3868003530 con 0x7f387c0a6030 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 >> v1:172.21.15.3:6793/0 conn(0x7f387c13f300 legacy=0x7f387c1416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 >> v1:172.21.15.203:6792/0 conn(0x7f387c0a2b10 legacy=0x7f387c13dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: _finish_hunting 0 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 monclient: found mon.noname-b 2024-05-21T17:21:44.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387c06cd30 con 0x7f387c0a6030 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: _finish_auth 0 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: _check_auth_tickets 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3874001bc0 con 0x7f387c0a6030 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 5 monclient: authenticate success, global_id 4385 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: dump: 2024-05-21T17:21:44.245 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3874001de0 con 0x7f387c0a6030 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 4 set_mon_vals no callback set 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 1 -- 172.21.15.3:0/565979352 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3874004f70 con 0x7f387c0a6030 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3880a81640 10 monclient: dump: 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 10 monclient: shutdown 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- 172.21.15.3:0/565979352 >> v1:172.21.15.3:6790/0 conn(0x7f387c0a6030 legacy=0x7f387c0a2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- 172.21.15.3:0/565979352 shutdown_connections 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- 172.21.15.3:0/565979352 >> 172.21.15.3:0/565979352 conn(0x7f387c0a93f0 msgr2=0x7f387c0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- 172.21.15.3:0/565979352 shutdown_connections 2024-05-21T17:21:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.228+0000 7f3883ce7640 1 -- 172.21.15.3:0/565979352 wait complete. 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: build_initial_monmap 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: monmap: 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:44.250 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 1 Processor -- start 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 1 -- start start 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: init 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:44.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: _add_conns ranks=[5,8,2,1,7,6,0,3,4] 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient(hunting): picked mon.noname-h con 0x7f387c0a2b10 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient(hunting): picked mon.noname-e con 0x7f387c0a6030 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient(hunting): picked mon.noname-b con 0x7f387c13f300 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f387c124d20 con 0x7f387c13f300 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f387c13bd80 con 0x7f387c0a2b10 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f387c127010 con 0x7f387c0a6030 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient(hunting): authenticate will time out at 971.393339s 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3882284640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f387c13f300 0x7f387c1d5df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52684/0 (socket says 172.21.15.3:52684) 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3882284640 1 -- 172.21.15.3:0/3867246397 learned_addr learned my addr 172.21.15.3:0/3867246397 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3881282640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f387c0a6030 0x7f387c1d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:52758/0 (socket says 172.21.15.3:52758) 2024-05-21T17:21:44.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4315792 0 0) 0x7f387c124d20 con 0x7f387c13f300 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): my global_id is 4394 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3860003530 con 0x7f387c13f300 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3943400993 0 0) 0x7f387c127010 con 0x7f387c0a6030 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): my global_id is 4427 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f387c124d20 con 0x7f387c0a6030 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3459456554 0 0) 0x7f387c124d20 con 0x7f387c0a6030 2024-05-21T17:21:44.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f387c127010 con 0x7f387c0a6030 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f386c001a40 con 0x7f387c0a6030 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): dump: 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3738586942 0 0) 0x7f3860003530 con 0x7f387c13f300 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f387c124d20 con 0x7f387c13f300 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3878001580 con 0x7f387c13f300 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient(hunting): dump: 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.256 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4222236076 0 0) 0x7f387c127010 con 0x7f387c0a6030 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 >> v1:172.21.15.203:6791/0 conn(0x7f387c0a2b10 legacy=0x7f387c13b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 >> v1:172.21.15.3:6790/0 conn(0x7f387c13f300 legacy=0x7f387c1d5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 monclient: found mon.e 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387c073f60 con 0x7f387c0a6030 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:44.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 5 monclient: authenticate success, global_id 4427 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: _renew_subs 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f387c126d30 con 0x7f387c0a6030 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: _renew_subs 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f387c1c43f0 con 0x7f387c0a6030 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f386c001d40 con 0x7f387c0a6030 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f386c003c10 con 0x7f387c0a6030 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3873fff640 4 set_mon_vals no callback set 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 10 monclient: dump: 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3864005090 con 0x7f387c0a6030 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f386c003db0 con 0x7f387c0a6030 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 2323575176 0 0) 0x7f386c045bf0 con 0x7f387c0a6030 2024-05-21T17:21:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.232+0000 7f3881a83640 10 monclient: get_auth_request con 0x7f3860039090 auth_method 0 2024-05-21T17:21:44.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.236+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f386c015d10 con 0x7f387c0a6030 2024-05-21T17:21:44.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.236+0000 7f38727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:44.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.236+0000 7f38727fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:44.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.472+0000 7f3883ce7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-05-21T17:21:44.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.472+0000 7f3883ce7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-05-21T17:21:44.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.472+0000 7f3883ce7640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:21:44.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.472+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3864005380 con 0x7f387c0a6030 2024-05-21T17:21:44.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.472+0000 7f38727fc640 1 -- 172.21.15.3:0/3867246397 <== mon.8 v1:172.21.15.3:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2715 (unknown 254746657 0 631582487) 0x7f386c015610 con 0x7f387c0a6030 2024-05-21T17:21:44.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.472+0000 7f38727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-05-21T17:21:44.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.472+0000 7f38727fc640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:44.479 INFO:teuthology.orchestra.run.smithi003.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","modified":"2024-05-21T17:20:49.979435Z","created":"2024-05-21T17:20:49.979435Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6790","nonce":0}]},"addr":"172.21.15.3:6790/0","public_addr":"172.21.15.3:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6790","nonce":0}]},"addr":"172.21.15.203:6790/0","public_addr":"172.21.15.203:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6791","nonce":0}]},"addr":"172.21.15.3:6791/0","public_addr":"172.21.15.3:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6791","nonce":0}]},"addr":"172.21.15.203:6791/0","public_addr":"172.21.15.203:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6792","nonce":0}]},"addr":"172.21.15.3:6792/0","public_addr":"172.21.15.3:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6792","nonce":0}]},"addr":"172.21.15.203:6792/0","public_addr":"172.21.15.203:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6793","nonce":0}]},"addr":"172.21.15.3:6793/0","public_addr":"172.21.15.3:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-21T17:21:44.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.476+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 >> v1:172.21.15.203:6812/343827626 conn(0x7f3860039090 legacy=0x7f386003b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.476+0000 7f3883ce7640 10 monclient: shutdown 2024-05-21T17:21:44.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.476+0000 7f3883ce7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:44.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.476+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 >> v1:172.21.15.3:6793/0 conn(0x7f387c0a6030 legacy=0x7f387c1d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.476+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 shutdown_connections 2024-05-21T17:21:44.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.476+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 >> 172.21.15.3:0/3867246397 conn(0x7f387c0a93f0 msgr2=0x7f387c0a5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:44.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.476+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 shutdown_connections 2024-05-21T17:21:44.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.476+0000 7f3883ce7640 1 -- 172.21.15.3:0/3867246397 wait complete. 2024-05-21T17:21:44.497 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-05-21T17:21:44.498 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-21T17:21:44.498 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-05-21T17:21:44.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.608+0000 7f5b8d558640 10 monclient: get_monmap_and_config 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.608+0000 7f5b8d558640 10 monclient: build_initial_monmap 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.608+0000 7f5b8d558640 10 monclient: monmap: 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:44.615271+0000 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:44.615271+0000 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:44.616 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.608+0000 7f5b8d558640 1 Processor -- start 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.608+0000 7f5b8d558640 1 -- start start 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.608+0000 7f5b8d558640 10 monclient: init 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: _add_conns ranks=[0,6,1,8,5,7,4,2,3] 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): picked mon.noname-a con 0x7f5b880a48c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): picked mon.noname-d con 0x7f5b880a4ca0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): picked mon.noname-f con 0x7f5b8813f210 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b880a88a0 con 0x7f5b880a48c0 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b880a6550 con 0x7f5b8813f210 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b880a28c0 con 0x7f5b880a4ca0 2024-05-21T17:21:44.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): authenticate will time out at 971.772682s 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b86ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5b880a48c0 0x7f5b880a5e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47634/0 (socket says 172.21.15.3:47634) 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b86ffd640 1 -- 172.21.15.3:0/2533199329 learned_addr learned my addr 172.21.15.3:0/2533199329 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b867fc640 1 --1- 172.21.15.3:0/2533199329 >> v1:172.21.15.3:6792/0 conn(0x7f5b880a4ca0 0x7f5b8813daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:50412/0 (socket says 172.21.15.3:50412) 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988923302 0 0) 0x7f5b880a28c0 con 0x7f5b880a4ca0 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient(hunting): my global_id is 4344 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b70003530 con 0x7f5b880a4ca0 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978791010 0 0) 0x7f5b880a88a0 con 0x7f5b880a48c0 2024-05-21T17:21:44.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient(hunting): my global_id is 4428 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b880a28c0 con 0x7f5b880a48c0 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 555677746 0 0) 0x7f5b70003530 con 0x7f5b880a4ca0 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 >> v1:172.21.15.203:6789/0 conn(0x7f5b8813f210 legacy=0x7f5b88141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 >> v1:172.21.15.3:6789/0 conn(0x7f5b880a48c0 legacy=0x7f5b880a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 monclient: found mon.noname-d 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b880a2ba0 con 0x7f5b880a4ca0 2024-05-21T17:21:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b7c001840 con 0x7f5b880a4ca0 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: dump: 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 5 monclient: authenticate success, global_id 4344 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b7c0019e0 con 0x7f5b880a4ca0 2024-05-21T17:21:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 4 set_mon_vals no callback set 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 1 -- 172.21.15.3:0/2533199329 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b7c004b30 con 0x7f5b880a4ca0 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b85ffb640 10 monclient: dump: 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: shutdown 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- 172.21.15.3:0/2533199329 >> v1:172.21.15.3:6792/0 conn(0x7f5b880a4ca0 legacy=0x7f5b8813daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- 172.21.15.3:0/2533199329 shutdown_connections 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- 172.21.15.3:0/2533199329 >> 172.21.15.3:0/2533199329 conn(0x7f5b880ab5b0 msgr2=0x7f5b880a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- 172.21.15.3:0/2533199329 shutdown_connections 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- 172.21.15.3:0/2533199329 wait complete. 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: build_initial_monmap 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: monmap: 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 Processor -- start 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- start start 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: init 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient: _add_conns ranks=[1,3,7,5,6,0,4,8,2] 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): picked mon.noname-f con 0x7f5b880a48c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): picked mon.noname-g con 0x7f5b880a4ca0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): picked mon.noname-i con 0x7f5b8813f210 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b880a5190 con 0x7f5b880a48c0 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b8809dab0 con 0x7f5b880a4ca0 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b880a0200 con 0x7f5b8813f210 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.612+0000 7f5b8d558640 10 monclient(hunting): authenticate will time out at 971.775773s 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b867fc640 1 --1- >> v1:172.21.15.203:6790/0 conn(0x7f5b880a4ca0 0x7f5b8809d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6790/0 says I am v1:172.21.15.3:45200/0 (socket says 172.21.15.3:45200) 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b867fc640 1 -- 172.21.15.3:0/1562056099 learned_addr learned my addr 172.21.15.3:0/1562056099 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 150438592 0 0) 0x7f5b880a5190 con 0x7f5b880a48c0 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): my global_id is 4411 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b74003530 con 0x7f5b880a48c0 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277214733 0 0) 0x7f5b8809dab0 con 0x7f5b880a4ca0 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): my global_id is 4449 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b880a5190 con 0x7f5b880a4ca0 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124469130 0 0) 0x7f5b880a0200 con 0x7f5b8813f210 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): my global_id is 4507 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b8809dab0 con 0x7f5b8813f210 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 242385212 0 0) 0x7f5b74003530 con 0x7f5b880a48c0 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b880a0200 con 0x7f5b880a48c0 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b78001660 con 0x7f5b880a48c0 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 588261318 0 0) 0x7f5b8809dab0 con 0x7f5b8813f210 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b74003530 con 0x7f5b8813f210 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b80001c00 con 0x7f5b8813f210 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 5654559 0 0) 0x7f5b880a5190 con 0x7f5b880a4ca0 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b8809dab0 con 0x7f5b880a4ca0 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b7c001440 con 0x7f5b880a4ca0 2024-05-21T17:21:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3288265582 0 0) 0x7f5b880a0200 con 0x7f5b880a48c0 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 >> v1:172.21.15.203:6792/0 conn(0x7f5b8813f210 legacy=0x7f5b881d5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 >> v1:172.21.15.203:6790/0 conn(0x7f5b880a4ca0 legacy=0x7f5b8809d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 monclient: found mon.f 2024-05-21T17:21:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.203:6789/0 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8809f060 con 0x7f5b880a48c0 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 5 monclient: authenticate success, global_id 4411 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 10 monclient: _renew_subs 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.203:6789/0 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8809f530 con 0x7f5b880a48c0 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 10 monclient: _renew_subs 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.203:6789/0 2024-05-21T17:21:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5b880a0ec0 con 0x7f5b880a48c0 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b78004aa0 con 0x7f5b880a48c0 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b857fa640 4 set_mon_vals no callback set 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b78004f90 con 0x7f5b880a48c0 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: dump: 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.644 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5b680065d0 con 0x7f5b68002870 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3582745103 0 0) 0x7f5b680065d0 con 0x7f5b68002870 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient(hunting): my global_id is 4437 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b880a0200 con 0x7f5b68002870 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1483665758 0 0) 0x7f5b880a0200 con 0x7f5b68002870 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b680065d0 con 0x7f5b68002870 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b7c001890 con 0x7f5b68002870 2024-05-21T17:21:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 10 monclient: dump: 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f5b78010110 con 0x7f5b880a48c0 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (unknown 2447148766 0 0) 0x7f5b78045be0 con 0x7f5b880a48c0 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 757101042 0 0) 0x7f5b680065d0 con 0x7f5b68002870 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.3:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b6800a170 con 0x7f5b68002870 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.616+0000 7f5b877fe640 10 monclient: get_auth_request con 0x7f5b74039760 auth_method 0 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.620+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5b7c005130 con 0x7f5b68002870 2024-05-21T17:21:44.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.620+0000 7f5b6f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:44.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.620+0000 7f5b6f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:44.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.620+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 >> v1:172.21.15.3:6789/0 conn(0x7f5b68002870 legacy=0x7f5b68004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.620+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 >> v1:172.21.15.3:6789/0 conn(0x7f5b68002870 legacy=0x7f5b68004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5b68009e40 con 0x7f5b68009120 2024-05-21T17:21:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 612707519 0 0) 0x7f5b68009e40 con 0x7f5b68009120 2024-05-21T17:21:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient(hunting): my global_id is 4446 2024-05-21T17:21:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b680065d0 con 0x7f5b68009120 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3964827042 0 0) 0x7f5b680065d0 con 0x7f5b68009120 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b68009e40 con 0x7f5b68009120 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b7c005960 con 0x7f5b68009120 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient: dump: 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4215894332 0 0) 0x7f5b68009e40 con 0x7f5b68009120 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 --> v1:172.21.15.3:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b6800c400 con 0x7f5b68009120 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 <== mon.0 v1:172.21.15.3:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7707 (unknown 0 0 2142443543) 0x7f5b7c007e80 con 0x7f5b68009120 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 >> v1:172.21.15.3:6789/0 conn(0x7f5b68009120 legacy=0x7f5b68009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b6f7fe640 1 -- 172.21.15.3:0/1562056099 >> v1:172.21.15.3:6789/0 conn(0x7f5b68009120 legacy=0x7f5b68009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "leader", 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 28559, 2024-05-21T17:21:44.656 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 18, 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:44.658 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:44.659 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:44.660 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:44.661 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.662 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.664 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.669 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.670 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.671 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.672 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.673 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:44.674 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.675 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:44.676 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:44.677 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 >> v1:172.21.15.203:6812/343827626 conn(0x7f5b74039760 legacy=0x7f5b7403bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 10 monclient: shutdown 2024-05-21T17:21:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 >> v1:172.21.15.203:6789/0 conn(0x7f5b880a48c0 legacy=0x7f5b8809cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b877fe640 1 -- 172.21.15.3:0/1562056099 reap_dead start 2024-05-21T17:21:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b877fe640 1 -- 172.21.15.3:0/1562056099 reap_dead start 2024-05-21T17:21:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 shutdown_connections 2024-05-21T17:21:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 >> 172.21.15.3:0/1562056099 conn(0x7f5b880ab5b0 msgr2=0x7f5b88141600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 shutdown_connections 2024-05-21T17:21:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.624+0000 7f5b8d558640 1 -- 172.21.15.3:0/1562056099 wait complete. 2024-05-21T17:21:44.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:44.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-05-21T17:21:44.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-21T17:21:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient: get_monmap_and_config 2024-05-21T17:21:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient: build_initial_monmap 2024-05-21T17:21:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient: monmap: 2024-05-21T17:21:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:44.813490+0000 2024-05-21T17:21:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:44.813490+0000 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:44.814 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 1 Processor -- start 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 1 -- start start 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient: init 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient: _add_conns ranks=[1,7,3,2,5,4,6,0,8] 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient(hunting): picked mon.noname-f con 0x7fd7380a5bd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient(hunting): picked mon.noname-i con 0x7fd7380a66c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient(hunting): picked mon.noname-g con 0x7fd7380a70c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd73809e480 con 0x7fd7380a5bd0 2024-05-21T17:21:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd73809cce0 con 0x7fd7380a70c0 2024-05-21T17:21:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd73809cfc0 con 0x7fd7380a66c0 2024-05-21T17:21:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.808+0000 7fd73f0e9640 10 monclient(hunting): authenticate will time out at 971.971914s 2024-05-21T17:21:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73d686640 1 --1- >> v1:172.21.15.203:6790/0 conn(0x7fd7380a70c0 0x7fd738141370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6790/0 says I am v1:172.21.15.3:45216/0 (socket says 172.21.15.3:45216) 2024-05-21T17:21:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73ce85640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd7380a5bd0 0x7fd7380a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.3:35926/0 (socket says 172.21.15.3:35926) 2024-05-21T17:21:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73d686640 1 -- 172.21.15.3:0/3580696450 learned_addr learned my addr 172.21.15.3:0/3580696450 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:44.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440352757 0 0) 0x7fd73809cfc0 con 0x7fd7380a66c0 2024-05-21T17:21:44.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): my global_id is 4516 2024-05-21T17:21:44.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd720003530 con 0x7fd7380a66c0 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2725407296 0 0) 0x7fd73809cce0 con 0x7fd7380a70c0 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): my global_id is 4458 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd73809cfc0 con 0x7fd7380a70c0 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314930679 0 0) 0x7fd73809e480 con 0x7fd7380a5bd0 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient(hunting): my global_id is 4420 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd73809cce0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3618485236 0 0) 0x7fd73809cce0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 >> v1:172.21.15.203:6792/0 conn(0x7fd7380a66c0 legacy=0x7fd73813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 >> v1:172.21.15.203:6790/0 conn(0x7fd7380a70c0 legacy=0x7fd738141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 monclient: found mon.noname-f 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.203:6789/0 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7380742e0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd734001760 con 0x7fd7380a5bd0 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: dump: 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 5 monclient: authenticate success, global_id 4420 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd734001980 con 0x7fd7380a5bd0 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 4 set_mon_vals no callback set 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 1 -- 172.21.15.3:0/3580696450 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd734004b30 con 0x7fd7380a5bd0 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd72f7fe640 10 monclient: dump: 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 10 monclient: shutdown 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 1 -- 172.21.15.3:0/3580696450 >> v1:172.21.15.203:6789/0 conn(0x7fd7380a5bd0 legacy=0x7fd7380a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 1 -- 172.21.15.3:0/3580696450 shutdown_connections 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 1 -- 172.21.15.3:0/3580696450 >> 172.21.15.3:0/3580696450 conn(0x7fd7380ab7d0 msgr2=0x7fd7380a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 1 -- 172.21.15.3:0/3580696450 shutdown_connections 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 1 -- 172.21.15.3:0/3580696450 wait complete. 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 10 monclient: build_initial_monmap 2024-05-21T17:21:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 10 monclient: monmap: 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:44.827 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:44.827 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 1 Processor -- start 2024-05-21T17:21:44.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 1 -- start start 2024-05-21T17:21:44.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.812+0000 7fd73f0e9640 10 monclient: init 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient: _add_conns ranks=[2,4,8,3,0,5,7,6,1] 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient(hunting): picked mon.noname-b con 0x7fd7380a5bd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient(hunting): picked mon.noname-c con 0x7fd7380a66c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient(hunting): picked mon.noname-e con 0x7fd7380a70c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7381351e0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7381376c0 con 0x7fd7380a66c0 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd738133fc0 con 0x7fd7380a70c0 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient(hunting): authenticate will time out at 971.976540s 2024-05-21T17:21:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73ce85640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fd7380a5bd0 0x7fd738136fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52696/0 (socket says 172.21.15.3:52696) 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73ce85640 1 -- 172.21.15.3:0/2037298102 learned_addr learned my addr 172.21.15.3:0/2037298102 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1725307792 0 0) 0x7fd7381351e0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): my global_id is 4403 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd724003530 con 0x7fd7380a5bd0 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3972514916 0 0) 0x7fd738133fc0 con 0x7fd7380a70c0 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): my global_id is 4436 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7381351e0 con 0x7fd7380a70c0 2024-05-21T17:21:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 558968957 0 0) 0x7fd7381376c0 con 0x7fd7380a66c0 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): my global_id is 4432 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd738133fc0 con 0x7fd7380a66c0 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2415855152 0 0) 0x7fd724003530 con 0x7fd7380a5bd0 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7381376c0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd7340016a0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): dump: 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1866531686 0 0) 0x7fd7381351e0 con 0x7fd7380a70c0 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd724003530 con 0x7fd7380a70c0 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd730001b90 con 0x7fd7380a70c0 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): dump: 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 751781188 0 0) 0x7fd738133fc0 con 0x7fd7380a66c0 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7381351e0 con 0x7fd7380a66c0 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 221513390 0 0) 0x7fd7381376c0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 >> v1:172.21.15.3:6793/0 conn(0x7fd7380a70c0 legacy=0x7fd7381d5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 >> v1:172.21.15.3:6791/0 conn(0x7fd7380a66c0 legacy=0x7fd7381c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: _finish_hunting 0 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 monclient: found mon.b 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd738133220 con 0x7fd7380a5bd0 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: _finish_auth 0 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: _check_auth_tickets 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 5 monclient: authenticate success, global_id 4403 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient: _renew_subs 2024-05-21T17:21:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd7381ca450 con 0x7fd7380a5bd0 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient: _renew_subs 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7381d6f10 con 0x7fd7380a5bd0 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7340018c0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd734005130 con 0x7fd7380a5bd0 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72effd640 4 set_mon_vals no callback set 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: dump: 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.839 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fd7340102b0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd7180065d0 con 0x7fd718002870 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (unknown 2447148766 0 0) 0x7fd734045db0 con 0x7fd7380a5bd0 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887193301 0 0) 0x7fd7180065d0 con 0x7fd718002870 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient(hunting): my global_id is 4412 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7381376c0 con 0x7fd718002870 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 130860904 0 0) 0x7fd7381376c0 con 0x7fd718002870 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7180065d0 con 0x7fd718002870 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd728003940 con 0x7fd718002870 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: dump: 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.842 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd73d686640 10 monclient: get_auth_request con 0x7fd724039590 auth_method 0 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3463377024 0 0) 0x7fd7180065d0 con 0x7fd718002870 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd71800a170 con 0x7fd718002870 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd728003ce0 con 0x7fd718002870 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 >> v1:172.21.15.3:6790/0 conn(0x7fd718002870 legacy=0x7fd718004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.816+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 >> v1:172.21.15.3:6790/0 conn(0x7fd718002870 legacy=0x7fd718004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd718009e30 con 0x7fd7180090f0 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797183173 0 0) 0x7fd718009e30 con 0x7fd7180090f0 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient(hunting): my global_id is 4421 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7180065d0 con 0x7fd7180090f0 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 521303241 0 0) 0x7fd7180065d0 con 0x7fd7180090f0 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd718009e30 con 0x7fd7180090f0 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd728004510 con 0x7fd7180090f0 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient: dump: 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:44.844 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:44.845 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3704852835 0 0) 0x7fd718009e30 con 0x7fd7180090f0 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 --> v1:172.21.15.3:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd71800c400 con 0x7fd7180090f0 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:44.846 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 28751, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 18, 2024-05-21T17:21:44.847 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:44.848 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:44.849 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:44.850 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:44.851 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:44.852 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.853 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:44.856 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.857 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.858 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:44.858 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:44.858 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.858 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.858 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:44.862 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.863 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.864 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:44.867 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:44.868 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:44.869 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:44.870 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:21:44.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:44.870 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:44.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:44.870 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:44.870 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 <== mon.2 v1:172.21.15.3:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7705 (unknown 0 0 961536233) 0x7fd728005940 con 0x7fd7180090f0 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 >> v1:172.21.15.3:6790/0 conn(0x7fd7180090f0 legacy=0x7fd718009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd72d7fa640 1 -- 172.21.15.3:0/2037298102 >> v1:172.21.15.3:6790/0 conn(0x7fd7180090f0 legacy=0x7fd718009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 >> v1:172.21.15.203:6812/343827626 conn(0x7fd724039590 legacy=0x7fd72403ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 10 monclient: shutdown 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 >> v1:172.21.15.3:6790/0 conn(0x7fd7380a5bd0 legacy=0x7fd738136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73d686640 1 -- 172.21.15.3:0/2037298102 reap_dead start 2024-05-21T17:21:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73d686640 1 -- 172.21.15.3:0/2037298102 reap_dead start 2024-05-21T17:21:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 shutdown_connections 2024-05-21T17:21:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 >> 172.21.15.3:0/2037298102 conn(0x7fd7380ab7d0 msgr2=0x7fd7380a18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 shutdown_connections 2024-05-21T17:21:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.824+0000 7fd73f0e9640 1 -- 172.21.15.3:0/2037298102 wait complete. 2024-05-21T17:21:44.873 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:44.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-05-21T17:21:44.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-21T17:21:44.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient: get_monmap_and_config 2024-05-21T17:21:44.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient: build_initial_monmap 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient: monmap: 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:44.988752+0000 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:44.988752+0000 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:44.989 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:44.990 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 Processor -- start 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- start start 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient: init 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient: _add_conns ranks=[2,0,5,8,6,7,1,3,4] 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient(hunting): picked mon.noname-b con 0x7f46b80a5bd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient(hunting): picked mon.noname-a con 0x7f46b80a66c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient(hunting): picked mon.noname-h con 0x7f46b80a70c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b809e480 con 0x7f46b80a66c0 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b809cce0 con 0x7f46b80a5bd0 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b809cfc0 con 0x7f46b80a70c0 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient(hunting): authenticate will time out at 972.146199s 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bdb12640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f46b80a5bd0 0x7f46b80a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52728/0 (socket says 172.21.15.3:52728) 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bdb12640 1 -- 172.21.15.3:0/3155953657 learned_addr learned my addr 172.21.15.3:0/3155953657 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bd311640 1 -- 172.21.15.3:0/3155953657 >> v1:172.21.15.3:6789/0 conn(0x7f46b80a66c0 legacy=0x7f46b813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bd311640 1 -- 172.21.15.3:0/3155953657 >> v1:172.21.15.3:6789/0 conn(0x7f46b80a66c0 legacy=0x7f46b813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bd311640 1 --1- 172.21.15.3:0/3155953657 >> v1:172.21.15.3:6789/0 conn(0x7f46b80a66c0 0x7f46b813dc40 :-1 s=OPENED pgs=77 cs=1 l=1).handle_message read tag failed 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bd311640 1 --1- 172.21.15.3:0/3155953657 >> v1:172.21.15.3:6789/0 conn(0x7f46b80a66c0 0x7f46b813dc40 :-1 s=OPENED pgs=77 cs=1 l=1).fault on lossy channel, failing 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.3:6789/0 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279559620 0 0) 0x7f46b809cce0 con 0x7f46b80a5bd0 2024-05-21T17:21:44.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient(hunting): my global_id is 4430 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a4003530 con 0x7f46b80a5bd0 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3906219949 0 0) 0x7f46a4003530 con 0x7f46b80a5bd0 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 >> v1:172.21.15.203:6791/0 conn(0x7f46b80a70c0 legacy=0x7f46b8141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 >> v1:172.21.15.3:6789/0 conn(0x7f46b80a66c0 legacy=0x7f46b813dc40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 monclient: found mon.noname-b 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b80742e0 con 0x7f46b80a5bd0 2024-05-21T17:21:45.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f46b4001840 con 0x7f46b80a5bd0 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 5 monclient: authenticate success, global_id 4430 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: dump: 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46b4001a60 con 0x7f46b80a5bd0 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 4 set_mon_vals no callback set 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 1 -- 172.21.15.3:0/3155953657 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f46b4001d50 con 0x7f46b80a5bd0 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bcb10640 10 monclient: dump: 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 10 monclient: shutdown 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- 172.21.15.3:0/3155953657 >> v1:172.21.15.3:6790/0 conn(0x7f46b80a5bd0 legacy=0x7f46b80a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- 172.21.15.3:0/3155953657 shutdown_connections 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- 172.21.15.3:0/3155953657 >> 172.21.15.3:0/3155953657 conn(0x7f46b80ab7d0 msgr2=0x7f46b80a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- 172.21.15.3:0/3155953657 shutdown_connections 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.984+0000 7f46bfd76640 1 -- 172.21.15.3:0/3155953657 wait complete. 2024-05-21T17:21:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: monmap: 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.005 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 1 Processor -- start 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 1 -- start start 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: init 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: _add_conns ranks=[3,5,2,7,0,4,6,1,8] 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient(hunting): picked mon.noname-g con 0x7f46b80a5bd0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient(hunting): picked mon.noname-h con 0x7f46b80a66c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient(hunting): picked mon.noname-b con 0x7f46b80a70c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b8142200 con 0x7f46b80a70c0 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b8122060 con 0x7f46b80a5bd0 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b81223b0 con 0x7f46b80a66c0 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient(hunting): authenticate will time out at 972.149194s 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46be313640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f46b80a70c0 0x7f46b81d5d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52730/0 (socket says 172.21.15.3:52730) 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46be313640 1 -- 172.21.15.3:0/816651647 learned_addr learned my addr 172.21.15.3:0/816651647 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4051257335 0 0) 0x7f46b8142200 con 0x7f46b80a70c0 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): my global_id is 4439 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f469c003530 con 0x7f46b80a70c0 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 867544321 0 0) 0x7f469c003530 con 0x7f46b80a70c0 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46b8142200 con 0x7f46b80a70c0 2024-05-21T17:21:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f46b0001de0 con 0x7f46b80a70c0 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): dump: 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.008 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037486992 0 0) 0x7f46b8122060 con 0x7f46b80a5bd0 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): my global_id is 4467 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f469c003530 con 0x7f46b80a5bd0 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889982961 0 0) 0x7f46b81223b0 con 0x7f46b80a66c0 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): my global_id is 4514 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b8122060 con 0x7f46b80a66c0 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 337833541 0 0) 0x7f46b8142200 con 0x7f46b80a70c0 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 >> v1:172.21.15.203:6791/0 conn(0x7f46b80a66c0 legacy=0x7f46b81d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 >> v1:172.21.15.203:6790/0 conn(0x7f46b80a5bd0 legacy=0x7f46b813c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 monclient: found mon.b 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b813c860 con 0x7f46b80a70c0 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 5 monclient: authenticate success, global_id 4439 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: _renew_subs 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46b8122660 con 0x7f46b80a70c0 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: _renew_subs 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f46b81c9f80 con 0x7f46b80a70c0 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46b0003c50 con 0x7f46b80a70c0 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f46b0004200 con 0x7f46b80a70c0 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46abfff640 4 set_mon_vals no callback set 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: dump: 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f46a00065d0 con 0x7f46a0002870 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f46b0004420 con 0x7f46b80a70c0 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (unknown 2447148766 0 0) 0x7f46b0045fd0 con 0x7f46b80a70c0 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3985457215 0 0) 0x7f46a00065d0 con 0x7f46a0002870 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient(hunting): my global_id is 4441 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b8142200 con 0x7f46a0002870 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2343584402 0 0) 0x7f46b8142200 con 0x7f46a0002870 2024-05-21T17:21:45.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46a00065d0 con 0x7f46a0002870 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f46b40014c0 con 0x7f46a0002870 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46bd311640 10 monclient: get_auth_request con 0x7f469c038e40 auth_method 0 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 10 monclient: dump: 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.014 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3235568475 0 0) 0x7f46a00065d0 con 0x7f46a0002870 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.988+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46a000a170 con 0x7f46a0002870 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.992+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f46b4001850 con 0x7f46a0002870 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.992+0000 7f46aa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.992+0000 7f46aa7fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.992+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 >> v1:172.21.15.3:6791/0 conn(0x7f46a0002870 legacy=0x7f46a0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.992+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 >> v1:172.21.15.3:6791/0 conn(0x7f46a0002870 legacy=0x7f46a0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f46a0009e80 con 0x7f46a0009140 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2517922632 0 0) 0x7f46a0009e80 con 0x7f46a0009140 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient(hunting): my global_id is 4450 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a00065d0 con 0x7f46a0009140 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 297754588 0 0) 0x7f46a00065d0 con 0x7f46a0009140 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46a0009e80 con 0x7f46a0009140 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f46b4001d40 con 0x7f46a0009140 2024-05-21T17:21:45.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient: dump: 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 678639931 0 0) 0x7f46a0009e80 con 0x7f46a0009140 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 --> v1:172.21.15.3:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46a000c420 con 0x7f46a0009140 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 <== mon.4 v1:172.21.15.3:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 457473528) 0x7f46b40073e0 con 0x7f46a0009140 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 >> v1:172.21.15.3:6791/0 conn(0x7f46a0009140 legacy=0x7f46a00095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46aa7fc640 1 -- 172.21.15.3:0/816651647 >> v1:172.21.15.3:6791/0 conn(0x7f46a0009140 legacy=0x7f46a00095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 28931, 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:45.020 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 18, 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.022 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.024 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.025 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.026 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:45.027 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.028 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.029 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.030 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.031 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.032 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:45.033 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.034 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.035 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.037 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:21:45.038 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.039 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.039 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:45.039 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:45.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 >> v1:172.21.15.203:6812/343827626 conn(0x7f469c038e40 legacy=0x7f469c03b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 10 monclient: shutdown 2024-05-21T17:21:45.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 >> v1:172.21.15.3:6790/0 conn(0x7f46b80a70c0 legacy=0x7f46b81d5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46be313640 1 -- 172.21.15.3:0/816651647 reap_dead start 2024-05-21T17:21:45.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46be313640 1 -- 172.21.15.3:0/816651647 reap_dead start 2024-05-21T17:21:45.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 shutdown_connections 2024-05-21T17:21:45.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 >> 172.21.15.3:0/816651647 conn(0x7f46b80ab7d0 msgr2=0x7f46b80a0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 shutdown_connections 2024-05-21T17:21:45.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:44.996+0000 7f46bfd76640 1 -- 172.21.15.3:0/816651647 wait complete. 2024-05-21T17:21:45.041 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:45.041 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-05-21T17:21:45.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient: get_monmap_and_config 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient: monmap: 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:45.168319+0000 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:45.168319+0000 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.169 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 Processor -- start 2024-05-21T17:21:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- start start 2024-05-21T17:21:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient: init 2024-05-21T17:21:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient: _add_conns ranks=[3,8,7,0,2,1,6,4,5] 2024-05-21T17:21:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient(hunting): picked mon.noname-g con 0x7f95980a49e0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient(hunting): picked mon.noname-e con 0x7f95980a54d0 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient(hunting): picked mon.noname-i con 0x7f95980a1030 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959806c920 con 0x7f95980a49e0 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959806e010 con 0x7f95980a1030 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959806ccf0 con 0x7f95980a54d0 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient(hunting): authenticate will time out at 972.325831s 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f95967fc640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f95980a54d0 0x7f959813df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:52766/0 (socket says 172.21.15.3:52766) 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f95967fc640 1 -- 172.21.15.3:0/156434381 learned_addr learned my addr 172.21.15.3:0/156434381 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2676389901 0 0) 0x7f959806ccf0 con 0x7f95980a54d0 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient(hunting): my global_id is 4445 2024-05-21T17:21:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9580003530 con 0x7f95980a54d0 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 746737665 0 0) 0x7f959806c920 con 0x7f95980a49e0 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient(hunting): my global_id is 4476 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f959806ccf0 con 0x7f95980a49e0 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1758899493 0 0) 0x7f9580003530 con 0x7f95980a54d0 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 >> v1:172.21.15.203:6792/0 conn(0x7f95980a1030 legacy=0x7f95981416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 >> v1:172.21.15.203:6790/0 conn(0x7f95980a49e0 legacy=0x7f95980a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 monclient: found mon.noname-e 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:21:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959809fe90 con 0x7f95980a54d0 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f958c001840 con 0x7f95980a54d0 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: dump: 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 5 monclient: authenticate success, global_id 4445 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f958c001a60 con 0x7f95980a54d0 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 1 -- 172.21.15.3:0/156434381 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f958c004bf0 con 0x7f95980a54d0 2024-05-21T17:21:45.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f9595ffb640 10 monclient: dump: 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.177 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 4 set_mon_vals no callback set 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 10 monclient: shutdown 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- 172.21.15.3:0/156434381 >> v1:172.21.15.3:6793/0 conn(0x7f95980a54d0 legacy=0x7f959813df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- 172.21.15.3:0/156434381 shutdown_connections 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- 172.21.15.3:0/156434381 >> 172.21.15.3:0/156434381 conn(0x7f95980a95d0 msgr2=0x7f95980a5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- 172.21.15.3:0/156434381 shutdown_connections 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.164+0000 7f959d505640 1 -- 172.21.15.3:0/156434381 wait complete. 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: monmap: 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.179 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.180 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.180 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.180 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.180 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.180 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 1 Processor -- start 2024-05-21T17:21:45.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 1 -- start start 2024-05-21T17:21:45.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: init 2024-05-21T17:21:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: _add_conns ranks=[4,0,6,3,7,2,5,1,8] 2024-05-21T17:21:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient(hunting): picked mon.noname-c con 0x7f95980a1030 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient(hunting): picked mon.noname-a con 0x7f95980a49e0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient(hunting): picked mon.noname-d con 0x7f95980a54d0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959809f5d0 con 0x7f95980a49e0 2024-05-21T17:21:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959813f510 con 0x7f95980a1030 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959813f7f0 con 0x7f95980a54d0 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient(hunting): authenticate will time out at 972.329030s 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f95967fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f95980a49e0 0x7f95981d28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47674/0 (socket says 172.21.15.3:47674) 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f95967fc640 1 -- 172.21.15.3:0/3154407590 learned_addr learned my addr 172.21.15.3:0/3154407590 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 303296980 0 0) 0x7f959809f5d0 con 0x7f95980a49e0 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): my global_id is 4455 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9584003530 con 0x7f95980a49e0 2024-05-21T17:21:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2152709299 0 0) 0x7f959813f510 con 0x7f95980a1030 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): my global_id is 4459 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f959809f5d0 con 0x7f95980a1030 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3848868457 0 0) 0x7f959813f7f0 con 0x7f95980a54d0 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): my global_id is 4353 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f959813f510 con 0x7f95980a54d0 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 390934656 0 0) 0x7f959809f5d0 con 0x7f95980a1030 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f959813f7f0 con 0x7f95980a1030 2024-05-21T17:21:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9588001ae0 con 0x7f95980a1030 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1826559032 0 0) 0x7f9584003530 con 0x7f95980a49e0 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f959809f5d0 con 0x7f95980a49e0 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f958c001c30 con 0x7f95980a49e0 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.190 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4174353355 0 0) 0x7f959813f510 con 0x7f95980a54d0 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9584003530 con 0x7f95980a54d0 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9590001c30 con 0x7f95980a54d0 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 527419776 0 0) 0x7f959813f7f0 con 0x7f95980a1030 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 >> v1:172.21.15.3:6792/0 conn(0x7f95980a54d0 legacy=0x7f95981d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 >> v1:172.21.15.3:6789/0 conn(0x7f95980a49e0 legacy=0x7f95981d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 monclient: found mon.c 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9598142a00 con 0x7f95980a1030 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 5 monclient: authenticate success, global_id 4459 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: _renew_subs 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f959813e820 con 0x7f95980a1030 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: _renew_subs 2024-05-21T17:21:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f95981ca760 con 0x7f95980a1030 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95880038d0 con 0x7f95980a1030 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f9588003dc0 con 0x7f95980a1030 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f95957fa640 4 set_mon_vals no callback set 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: dump: 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f95780065d0 con 0x7f9578002870 2024-05-21T17:21:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f958800ef40 con 0x7f95980a1030 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (unknown 2447148766 0 0) 0x7f9588044a90 con 0x7f95980a1030 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2302787508 0 0) 0x7f95780065d0 con 0x7f9578002870 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient(hunting): my global_id is 4362 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f959813f7f0 con 0x7f9578002870 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 694518848 0 0) 0x7f959813f7f0 con 0x7f9578002870 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95780065d0 con 0x7f9578002870 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f958c004e70 con 0x7f9578002870 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 10 monclient: dump: 2024-05-21T17:21:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f95977fe640 10 monclient: get_auth_request con 0x7f95840396b0 auth_method 0 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 526631955 0 0) 0x7f95780065d0 con 0x7f9578002870 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.168+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f957800a170 con 0x7f9578002870 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.172+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f958c007480 con 0x7f9578002870 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.172+0000 7f957f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.172+0000 7f957f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.172+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 >> v1:172.21.15.3:6792/0 conn(0x7f9578002870 legacy=0x7f9578004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.172+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 >> v1:172.21.15.3:6792/0 conn(0x7f9578002870 legacy=0x7f9578004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9578009e10 con 0x7f95780090f0 2024-05-21T17:21:45.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1423022632 0 0) 0x7f9578009e10 con 0x7f95780090f0 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient(hunting): my global_id is 4371 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95780065d0 con 0x7f95780090f0 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 694174450 0 0) 0x7f95780065d0 con 0x7f95780090f0 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9578009e10 con 0x7f95780090f0 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f958c005670 con 0x7f95780090f0 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient: dump: 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.199 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 558865516 0 0) 0x7f9578009e10 con 0x7f95780090f0 2024-05-21T17:21:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 --> v1:172.21.15.3:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f957800c3b0 con 0x7f95780090f0 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 <== mon.6 v1:172.21.15.3:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 2338495466) 0x7f958c006b60 con 0x7f95780090f0 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 >> v1:172.21.15.3:6792/0 conn(0x7f95780090f0 legacy=0x7f9578009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f957f7fe640 1 -- 172.21.15.3:0/3154407590 >> v1:172.21.15.3:6792/0 conn(0x7f95780090f0 legacy=0x7f9578009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:21:45.201 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 29099, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.202 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 18, 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.203 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:45.205 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.206 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.207 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.208 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.209 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.210 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.211 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.213 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:45.214 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:45.214 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.214 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.214 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:45.214 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:45.215 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.216 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.217 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.218 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.219 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 >> v1:172.21.15.203:6812/343827626 conn(0x7f95840396b0 legacy=0x7f958403bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 10 monclient: shutdown 2024-05-21T17:21:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 >> v1:172.21.15.3:6791/0 conn(0x7f95980a1030 legacy=0x7f95981422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f95977fe640 1 -- 172.21.15.3:0/3154407590 reap_dead start 2024-05-21T17:21:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f95977fe640 1 -- 172.21.15.3:0/3154407590 reap_dead start 2024-05-21T17:21:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 shutdown_connections 2024-05-21T17:21:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 >> 172.21.15.3:0/3154407590 conn(0x7f95980a95d0 msgr2=0x7f95980a7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 shutdown_connections 2024-05-21T17:21:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.176+0000 7f959d505640 1 -- 172.21.15.3:0/3154407590 wait complete. 2024-05-21T17:21:45.224 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:45.224 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-05-21T17:21:45.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-05-21T17:21:45.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.344+0000 7fedc90b8640 10 monclient: get_monmap_and_config 2024-05-21T17:21:45.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.344+0000 7fedc90b8640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.344+0000 7fedc90b8640 10 monclient: monmap: 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:45.351736+0000 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:45.351736+0000 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.353 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.353 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.353 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.353 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.353 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.353 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 Processor -- start 2024-05-21T17:21:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- start start 2024-05-21T17:21:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient: init 2024-05-21T17:21:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient: _add_conns ranks=[2,1,0,8,3,5,4,6,7] 2024-05-21T17:21:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient(hunting): picked mon.noname-b con 0x7fedc40a5bd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient(hunting): picked mon.noname-f con 0x7fedc40a66c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient(hunting): picked mon.noname-a con 0x7fedc40a70c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc409e480 con 0x7fedc40a70c0 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc409cce0 con 0x7fedc40a66c0 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc409cfc0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient(hunting): authenticate will time out at 972.509092s 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc2d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fedc40a5bd0 0x7fedc40a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52746/0 (socket says 172.21.15.3:52746) 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc359e640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fedc40a70c0 0x7fedc4141370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47678/0 (socket says 172.21.15.3:47678) 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc2d9d640 1 -- 172.21.15.3:0/1676642450 learned_addr learned my addr 172.21.15.3:0/1676642450 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3207006910 0 0) 0x7fedc409cfc0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): my global_id is 4448 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedb0003530 con 0x7fedc40a5bd0 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584857815 0 0) 0x7fedc409e480 con 0x7fedc40a70c0 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): my global_id is 4464 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedc409cfc0 con 0x7fedc40a70c0 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2441407074 0 0) 0x7fedc409cce0 con 0x7fedc40a66c0 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient(hunting): my global_id is 4429 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedc409e480 con 0x7fedc40a66c0 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1878354118 0 0) 0x7fedb0003530 con 0x7fedc40a5bd0 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 >> v1:172.21.15.203:6789/0 conn(0x7fedc40a66c0 legacy=0x7fedc413dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 >> v1:172.21.15.3:6789/0 conn(0x7fedc40a70c0 legacy=0x7fedc4141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 monclient: found mon.noname-b 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc40742e0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fedb4001840 con 0x7fedc40a5bd0 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: dump: 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 5 monclient: authenticate success, global_id 4448 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedb40019e0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 1 -- 172.21.15.3:0/1676642450 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fedb4004bf0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc259c640 10 monclient: dump: 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 4 set_mon_vals no callback set 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient: shutdown 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- 172.21.15.3:0/1676642450 >> v1:172.21.15.3:6790/0 conn(0x7fedc40a5bd0 legacy=0x7fedc40a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- 172.21.15.3:0/1676642450 shutdown_connections 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- 172.21.15.3:0/1676642450 >> 172.21.15.3:0/1676642450 conn(0x7fedc40ab7d0 msgr2=0x7fedc40a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- 172.21.15.3:0/1676642450 shutdown_connections 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 -- 172.21.15.3:0/1676642450 wait complete. 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 10 monclient: monmap: 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.348+0000 7fedc90b8640 1 Processor -- start 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 1 -- start start 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient: init 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient: _add_conns ranks=[6,2,5,1,4,3,0,7,8] 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient(hunting): picked mon.noname-d con 0x7fedc40a5bd0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient(hunting): picked mon.noname-b con 0x7fedc40a66c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient(hunting): picked mon.noname-h con 0x7fedc40a70c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc4140650 con 0x7fedc40a66c0 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc4141300 con 0x7fedc40a70c0 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc41cabd0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient(hunting): authenticate will time out at 972.512634s 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc2d9d640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7fedc40a5bd0 0x7fedc41c9db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:50460/0 (socket says 172.21.15.3:50460) 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc2d9d640 1 -- 172.21.15.3:0/2420019183 learned_addr learned my addr 172.21.15.3:0/2420019183 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204866079 0 0) 0x7fedc41cabd0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): my global_id is 4380 2024-05-21T17:21:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda4003530 con 0x7fedc40a5bd0 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2317166989 0 0) 0x7fedc409e480 con 0x7fedc40a66c0 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): my global_id is 4457 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedc41cabd0 con 0x7fedc40a66c0 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130616600 0 0) 0x7fedbc0014c0 con 0x7fedc40a70c0 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): my global_id is 4523 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedc409e480 con 0x7fedc40a70c0 2024-05-21T17:21:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3005375219 0 0) 0x7feda4003530 con 0x7fedc40a5bd0 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedbc0014c0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fedb4004b30 con 0x7fedc40a5bd0 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1449541321 0 0) 0x7fedc41cabd0 con 0x7fedc40a66c0 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feda4003530 con 0x7fedc40a66c0 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fedac001a20 con 0x7fedc40a66c0 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): dump: 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 245811601 0 0) 0x7fedbc0014c0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 >> v1:172.21.15.203:6791/0 conn(0x7fedc40a70c0 legacy=0x7fedc41d58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 >> v1:172.21.15.3:6790/0 conn(0x7fedc40a66c0 legacy=0x7fedc41ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 monclient: found mon.d 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc40a79f0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 5 monclient: authenticate success, global_id 4380 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient: _renew_subs 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fedc409e6a0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient: _renew_subs 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fedc4141ba0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedb40018c0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fedb40051f0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc1d9b640 4 set_mon_vals no callback set 2024-05-21T17:21:45.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: dump: 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.378 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feda80065d0 con 0x7feda8002870 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fedb4010370 con 0x7fedc40a5bd0 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7fedb4045dd0 con 0x7fedc40a5bd0 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4154375815 0 0) 0x7feda80065d0 con 0x7feda8002870 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient(hunting): my global_id is 4454 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedbc0014c0 con 0x7feda8002870 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedc359e640 10 monclient: get_auth_request con 0x7feda4039470 auth_method 0 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4204465613 0 0) 0x7fedbc0014c0 con 0x7feda8002870 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feda80065d0 con 0x7feda8002870 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fedac003960 con 0x7feda8002870 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 10 monclient: dump: 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.382 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2202630767 0 0) 0x7feda80065d0 con 0x7feda8002870 2024-05-21T17:21:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.352+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feda800a170 con 0x7feda8002870 2024-05-21T17:21:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.356+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fedac006480 con 0x7feda8002870 2024-05-21T17:21:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.356+0000 7fedbb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.356+0000 7fedbb7fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.356+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 >> v1:172.21.15.3:6793/0 conn(0x7feda8002870 legacy=0x7feda8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.356+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 >> v1:172.21.15.3:6793/0 conn(0x7feda8002870 legacy=0x7feda8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.360+0000 7fedc90b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.360+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feda8009e60 con 0x7feda8009120 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.360+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656922117 0 0) 0x7feda8009e60 con 0x7feda8009120 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.360+0000 7fedbb7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.360+0000 7fedbb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.360+0000 7fedbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.360+0000 7fedbb7fe640 10 monclient(hunting): my global_id is 4463 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.360+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda80065d0 con 0x7feda8009120 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 875372413 0 0) 0x7feda80065d0 con 0x7feda8009120 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feda8009e60 con 0x7feda8009120 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fedac004630 con 0x7feda8009120 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 10 monclient: dump: 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 949811221 0 0) 0x7feda8009e60 con 0x7feda8009120 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 --> v1:172.21.15.3:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feda800c3b0 con 0x7feda8009120 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:45.386 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 29295, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:45.387 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 18, 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.388 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.389 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:45.390 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:45.391 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.392 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:45.393 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:45.394 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.396 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.397 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.399 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.401 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.402 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:45.404 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.405 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:45.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.407 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.407 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.407 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:45.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.407 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.407 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:45.407 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:45.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 <== mon.8 v1:172.21.15.3:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 1990747787) 0x7fedac005b40 con 0x7feda8009120 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 >> v1:172.21.15.3:6793/0 conn(0x7feda8009120 legacy=0x7feda80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedbb7fe640 1 -- 172.21.15.3:0/2420019183 >> v1:172.21.15.3:6793/0 conn(0x7feda8009120 legacy=0x7feda80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 >> v1:172.21.15.203:6812/343827626 conn(0x7feda4039470 legacy=0x7feda403b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc90b8640 10 monclient: shutdown 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc90b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 >> v1:172.21.15.3:6792/0 conn(0x7fedc40a5bd0 legacy=0x7fedc41c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc359e640 1 -- 172.21.15.3:0/2420019183 reap_dead start 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc359e640 1 -- 172.21.15.3:0/2420019183 reap_dead start 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 shutdown_connections 2024-05-21T17:21:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 >> 172.21.15.3:0/2420019183 conn(0x7fedc40ab7d0 msgr2=0x7fedc4075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 shutdown_connections 2024-05-21T17:21:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.364+0000 7fedc90b8640 1 -- 172.21.15.3:0/2420019183 wait complete. 2024-05-21T17:21:45.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:45.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-05-21T17:21:45.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-05-21T17:21:45.458 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[26982]: starting. 2024-05-21T17:21:45.459 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[26982]: creating pool ceph_test_rados_delete_pools_parallel.smithi203-26981 2024-05-21T17:21:45.459 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[26982]: created object 0... 2024-05-21T17:21:45.459 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[26982]: created object 25... 2024-05-21T17:21:45.459 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[26982]: created object 49... 2024-05-21T17:21:45.459 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[26982]: finishing. 2024-05-21T17:21:45.459 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[26982]: shutting down. 2024-05-21T17:21:45.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient: get_monmap_and_config 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient: monmap: 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:45.528452+0000 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:45.528452+0000 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.529 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 Processor -- start 2024-05-21T17:21:45.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- start start 2024-05-21T17:21:45.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient: init 2024-05-21T17:21:45.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient: _add_conns ranks=[7,0,3,8,5,1,4,2,6] 2024-05-21T17:21:45.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient(hunting): picked mon.noname-i con 0x7f6f900a1c40 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:45.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient(hunting): picked mon.noname-a con 0x7f6f900a0fc0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:45.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient(hunting): picked mon.noname-g con 0x7f6f900a0a60 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:45.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f900a2730 con 0x7f6f900a0fc0 2024-05-21T17:21:45.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f90073a10 con 0x7f6f900a0a60 2024-05-21T17:21:45.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f900a2bf0 con 0x7f6f900a1c40 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient(hunting): authenticate will time out at 972.685854s 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f95d1c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6f900a0fc0 0x7f6f9013b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47688/0 (socket says 172.21.15.3:47688) 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f95d1c640 1 -- 172.21.15.3:0/3608969114 learned_addr learned my addr 172.21.15.3:0/3608969114 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4116292997 0 0) 0x7f6f900a2730 con 0x7f6f900a0fc0 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient(hunting): my global_id is 4473 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f7c003530 con 0x7f6f900a0fc0 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3821670124 0 0) 0x7f6f7c003530 con 0x7f6f900a0fc0 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 >> v1:172.21.15.203:6792/0 conn(0x7f6f900a1c40 legacy=0x7f6f900a2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 >> v1:172.21.15.203:6790/0 conn(0x7f6f900a0a60 legacy=0x7f6f9013f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 monclient: found mon.noname-a 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f900a2ed0 con 0x7f6f900a0fc0 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6f80001840 con 0x7f6f900a0fc0 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 5 monclient: authenticate success, global_id 4473 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:45.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: dump: 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.541 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f800019e0 con 0x7f6f900a0fc0 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 1 -- 172.21.15.3:0/3608969114 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6f80004bf0 con 0x7f6f900a0fc0 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f9551b640 10 monclient: dump: 2024-05-21T17:21:45.542 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 4 set_mon_vals no callback set 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 10 monclient: shutdown 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- 172.21.15.3:0/3608969114 >> v1:172.21.15.3:6789/0 conn(0x7f6f900a0fc0 legacy=0x7f6f9013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- 172.21.15.3:0/3608969114 shutdown_connections 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- 172.21.15.3:0/3608969114 >> 172.21.15.3:0/3608969114 conn(0x7f6f900a72c0 msgr2=0x7f6f900a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- 172.21.15.3:0/3608969114 shutdown_connections 2024-05-21T17:21:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.524+0000 7f6f98781640 1 -- 172.21.15.3:0/3608969114 wait complete. 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: monmap: 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 1 Processor -- start 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 1 -- start start 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: init 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: _add_conns ranks=[8,3,4,2,0,6,5,1,7] 2024-05-21T17:21:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient(hunting): picked mon.noname-e con 0x7f6f900a0a60 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient(hunting): picked mon.noname-g con 0x7f6f900a1c40 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient(hunting): picked mon.noname-c con 0x7f6f901402d0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9013f550 con 0x7f6f900a1c40 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9011e430 con 0x7f6f901402d0 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9011db90 con 0x7f6f900a0a60 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient(hunting): authenticate will time out at 972.689182s 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f9651d640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f6f900a0a60 0x7f6f90136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:52792/0 (socket says 172.21.15.3:52792) 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f9651d640 1 -- 172.21.15.3:0/817274836 learned_addr learned my addr 172.21.15.3:0/817274836 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1095677800 0 0) 0x7f6f9011e430 con 0x7f6f901402d0 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): my global_id is 4468 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f74003530 con 0x7f6f901402d0 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780810920 0 0) 0x7f6f9011db90 con 0x7f6f900a0a60 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): my global_id is 4472 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f9011e430 con 0x7f6f900a0a60 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461615743 0 0) 0x7f6f9013f550 con 0x7f6f900a1c40 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): my global_id is 4494 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f9011db90 con 0x7f6f900a1c40 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2827458669 0 0) 0x7f6f74003530 con 0x7f6f901402d0 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f9013f550 con 0x7f6f901402d0 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6f88001e60 con 0x7f6f901402d0 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.550 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2441087513 0 0) 0x7f6f9011e430 con 0x7f6f900a0a60 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f74003530 con 0x7f6f900a0a60 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6f8c001980 con 0x7f6f900a0a60 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3385290441 0 0) 0x7f6f9013f550 con 0x7f6f901402d0 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 >> v1:172.21.15.3:6793/0 conn(0x7f6f900a0a60 legacy=0x7f6f90136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 >> v1:172.21.15.203:6790/0 conn(0x7f6f900a1c40 legacy=0x7f6f901d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 monclient: found mon.c 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f901371e0 con 0x7f6f901402d0 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 5 monclient: authenticate success, global_id 4468 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: _renew_subs 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f901c7a30 con 0x7f6f901402d0 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: _renew_subs 2024-05-21T17:21:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f901d50e0 con 0x7f6f901402d0 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f88002000 con 0x7f6f901402d0 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6f88003f90 con 0x7f6f901402d0 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f94d1a640 4 set_mon_vals no callback set 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 10 monclient: dump: 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6f780065d0 con 0x7f6f78002870 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f6f8800f110 con 0x7f6f901402d0 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7f6f88044d80 con 0x7f6f901402d0 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.528+0000 7f6f95d1c640 10 monclient: get_auth_request con 0x7f6f74039590 auth_method 0 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3012595427 0 0) 0x7f6f780065d0 con 0x7f6f78002870 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient(hunting): my global_id is 4438 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f9013f550 con 0x7f6f78002870 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 935578173 0 0) 0x7f6f9013f550 con 0x7f6f78002870 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f780065d0 con 0x7f6f78002870 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6f8c003960 con 0x7f6f78002870 2024-05-21T17:21:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient: dump: 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 649571880 0 0) 0x7f6f780065d0 con 0x7f6f78002870 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f7800a170 con 0x7f6f78002870 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6f8c0057b0 con 0x7f6f78002870 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 >> v1:172.21.15.203:6789/0 conn(0x7f6f78002870 legacy=0x7f6f78004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.532+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 >> v1:172.21.15.203:6789/0 conn(0x7f6f78002870 legacy=0x7f6f78004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.536+0000 7f6f98781640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.536+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6f78009e10 con 0x7f6f780090f0 2024-05-21T17:21:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016390370 0 0) 0x7f6f78009e10 con 0x7f6f780090f0 2024-05-21T17:21:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient(hunting): my global_id is 4447 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f780065d0 con 0x7f6f780090f0 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3572541508 0 0) 0x7f6f780065d0 con 0x7f6f780090f0 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f78009e10 con 0x7f6f780090f0 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6f8c004260 con 0x7f6f780090f0 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient: dump: 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.562 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2711766255 0 0) 0x7f6f78009e10 con 0x7f6f780090f0 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6f7800c3b0 con 0x7f6f780090f0 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 2548601431) 0x7f6f8c0042d0 con 0x7f6f780090f0 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 >> v1:172.21.15.203:6789/0 conn(0x7f6f780090f0 legacy=0x7f6f78009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f86ffd640 1 -- 172.21.15.3:0/817274836 >> v1:172.21.15.203:6789/0 conn(0x7f6f780090f0 legacy=0x7f6f78009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:45.563 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 29451, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:45.564 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 19, 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.565 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.566 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:45.567 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.568 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.569 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.570 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:45.573 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.574 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:45.576 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.578 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.579 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.581 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.583 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 3 2024-05-21T17:21:45.583 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.583 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.583 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:45.583 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:45.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 >> v1:172.21.15.203:6812/343827626 conn(0x7f6f74039590 legacy=0x7f6f7403ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f98781640 10 monclient: shutdown 2024-05-21T17:21:45.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f98781640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 >> v1:172.21.15.3:6791/0 conn(0x7f6f901402d0 legacy=0x7f6f901d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f96d1e640 1 -- 172.21.15.3:0/817274836 reap_dead start 2024-05-21T17:21:45.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f96d1e640 1 -- 172.21.15.3:0/817274836 reap_dead start 2024-05-21T17:21:45.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 shutdown_connections 2024-05-21T17:21:45.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 >> 172.21.15.3:0/817274836 conn(0x7f6f900a72c0 msgr2=0x7f6f900a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 shutdown_connections 2024-05-21T17:21:45.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.540+0000 7f6f98781640 1 -- 172.21.15.3:0/817274836 wait complete. 2024-05-21T17:21:45.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:45.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-05-21T17:21:45.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-05-21T17:21:45.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient: get_monmap_and_config 2024-05-21T17:21:45.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient: monmap: 2024-05-21T17:21:45.701 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.701 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:45.701338+0000 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:45.701338+0000 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.702 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 1 Processor -- start 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 1 -- start start 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient: init 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient: _add_conns ranks=[3,4,0,8,5,7,1,6,2] 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient(hunting): picked mon.noname-g con 0x7f71cc0a5bd0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient(hunting): picked mon.noname-c con 0x7f71cc0a66c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient(hunting): picked mon.noname-a con 0x7f71cc0a70c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71cc09e480 con 0x7f71cc0a70c0 2024-05-21T17:21:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71cc09cce0 con 0x7f71cc0a5bd0 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71cc09cfc0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 10 monclient(hunting): authenticate will time out at 972.858884s 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d1374640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f71cc0a66c0 0x7f71cc13dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:38614/0 (socket says 172.21.15.3:38614) 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d1374640 1 -- 172.21.15.3:0/1611008609 learned_addr learned my addr 172.21.15.3:0/1611008609 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 560107052 0 0) 0x7f71cc09cfc0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient(hunting): my global_id is 4477 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b8003530 con 0x7f71cc0a66c0 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 638466837 0 0) 0x7f71cc09e480 con 0x7f71cc0a70c0 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient(hunting): my global_id is 4482 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71cc09cfc0 con 0x7f71cc0a70c0 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2102593113 0 0) 0x7f71b8003530 con 0x7f71cc0a66c0 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 >> v1:172.21.15.203:6790/0 conn(0x7f71cc0a5bd0 legacy=0x7f71cc0a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 >> v1:172.21.15.3:6789/0 conn(0x7f71cc0a70c0 legacy=0x7f71cc141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 monclient: found mon.noname-c 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc0742e0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f71c8001840 con 0x7f71cc0a66c0 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d0b73640 10 monclient: dump: 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 5 monclient: authenticate success, global_id 4477 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.696+0000 7f71d3dd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71c80019e0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d0b73640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d0b73640 1 -- 172.21.15.3:0/1611008609 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f71c8004bf0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d0b73640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d0b73640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d0b73640 10 monclient: dump: 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 4 set_mon_vals no callback set 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: shutdown 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1611008609 >> v1:172.21.15.3:6791/0 conn(0x7f71cc0a66c0 legacy=0x7f71cc13dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1611008609 shutdown_connections 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1611008609 >> 172.21.15.3:0/1611008609 conn(0x7f71cc0ab7d0 msgr2=0x7f71cc0a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1611008609 shutdown_connections 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1611008609 wait complete. 2024-05-21T17:21:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: monmap: 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 Processor -- start 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- start start 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: init 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: _add_conns ranks=[7,4,3,6,1,5,2,0,8] 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient(hunting): picked mon.noname-i con 0x7f71cc0a5bd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient(hunting): picked mon.noname-c con 0x7f71cc0a66c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient(hunting): picked mon.noname-g con 0x7f71cc0a70c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71cc09e6a0 con 0x7f71cc0a70c0 2024-05-21T17:21:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71cc13d7e0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71cc13f7f0 con 0x7f71cc0a5bd0 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient(hunting): authenticate will time out at 972.862199s 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d1374640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f71cc0a66c0 0x7f71cc1d2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:38616/0 (socket says 172.21.15.3:38616) 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d1374640 1 -- 172.21.15.3:0/1000404802 learned_addr learned my addr 172.21.15.3:0/1000404802 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2772691640 0 0) 0x7f71cc13d7e0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient(hunting): my global_id is 4486 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b0003530 con 0x7f71cc0a66c0 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2212365621 0 0) 0x7f71b0003530 con 0x7f71cc0a66c0 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71cc13d7e0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f71c80047f0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient(hunting): dump: 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1757503687 0 0) 0x7f71cc13d7e0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 >> v1:172.21.15.203:6792/0 conn(0x7f71cc0a5bd0 legacy=0x7f71cc142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 >> v1:172.21.15.203:6790/0 conn(0x7f71cc0a70c0 legacy=0x7f71cc1d5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 monclient: found mon.c 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc142720 con 0x7f71cc0a66c0 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 5 monclient: authenticate success, global_id 4486 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: _renew_subs 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71cc13df90 con 0x7f71cc0a66c0 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: _renew_subs 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:21:45.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f71cc13e750 con 0x7f71cc0a66c0 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71c80011c0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f71c8001940 con 0x7f71cc0a66c0 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71bffff640 4 set_mon_vals no callback set 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 10 monclient: dump: 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.722 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71b40065d0 con 0x7f71b4002870 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.700+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f71c80101d0 con 0x7f71cc0a66c0 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7f71c8045e50 con 0x7f71cc0a66c0 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71d2376640 10 monclient: get_auth_request con 0x7f71b0038b40 auth_method 0 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2733223713 0 0) 0x7f71b40065d0 con 0x7f71b4002870 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient(hunting): my global_id is 4521 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71cc13d7e0 con 0x7f71b4002870 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2324062946 0 0) 0x7f71cc13d7e0 con 0x7f71b4002870 2024-05-21T17:21:45.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71b40065d0 con 0x7f71b4002870 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f71c00019b0 con 0x7f71b4002870 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient: dump: 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1636360510 0 0) 0x7f71b40065d0 con 0x7f71b4002870 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.203:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71b400a170 con 0x7f71b4002870 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f71c0001ca0 con 0x7f71b4002870 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 >> v1:172.21.15.203:6790/0 conn(0x7f71b4002870 legacy=0x7f71b4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.704+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 >> v1:172.21.15.203:6790/0 conn(0x7f71b4002870 legacy=0x7f71b4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71b4009e80 con 0x7f71b4009140 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388093117 0 0) 0x7f71b4009e80 con 0x7f71b4009140 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient(hunting): my global_id is 4530 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b40065d0 con 0x7f71b4009140 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3237526099 0 0) 0x7f71b40065d0 con 0x7f71b4009140 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71b4009e80 con 0x7f71b4009140 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f71c0003b30 con 0x7f71b4009140 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient: dump: 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4020279080 0 0) 0x7f71b4009e80 con 0x7f71b4009140 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 --> v1:172.21.15.203:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71b400c420 con 0x7f71b4009140 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 <== mon.3 v1:172.21.15.203:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 1764411254) 0x7f71c0004fc0 con 0x7f71b4009140 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 >> v1:172.21.15.203:6790/0 conn(0x7f71b4009140 legacy=0x7f71b40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71be7fc640 1 -- 172.21.15.3:0/1000404802 >> v1:172.21.15.203:6790/0 conn(0x7f71b4009140 legacy=0x7f71b40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.729 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 29619, 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 19, 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:45.742 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:45.749 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:45.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 >> v1:172.21.15.203:6812/343827626 conn(0x7f71b0038b40 legacy=0x7f71b003b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 10 monclient: shutdown 2024-05-21T17:21:45.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 >> v1:172.21.15.3:6791/0 conn(0x7f71cc0a66c0 legacy=0x7f71cc1d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d2376640 1 -- 172.21.15.3:0/1000404802 reap_dead start 2024-05-21T17:21:45.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d2376640 1 -- 172.21.15.3:0/1000404802 reap_dead start 2024-05-21T17:21:45.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 shutdown_connections 2024-05-21T17:21:45.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 >> 172.21.15.3:0/1000404802 conn(0x7f71cc0ab7d0 msgr2=0x7f71cc139b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 shutdown_connections 2024-05-21T17:21:45.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.712+0000 7f71d3dd9640 1 -- 172.21.15.3:0/1000404802 wait complete. 2024-05-21T17:21:45.751 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:45.752 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-05-21T17:21:45.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-05-21T17:21:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient: get_monmap_and_config 2024-05-21T17:21:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient: monmap: 2024-05-21T17:21:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:45.868402+0000 2024-05-21T17:21:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:45.868402+0000 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.869 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 Processor -- start 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- start start 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient: init 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient: _add_conns ranks=[2,7,1,5,0,6,8,4,3] 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient(hunting): picked mon.noname-b con 0x7f80e80a5bd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient(hunting): picked mon.noname-i con 0x7f80e80a66c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient(hunting): picked mon.noname-f con 0x7f80e80a70c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e809e480 con 0x7f80e80a70c0 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e809cce0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e809cfc0 con 0x7f80e80a66c0 2024-05-21T17:21:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient(hunting): authenticate will time out at 973.025830s 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e6ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f80e80a5bd0 0x7f80e80a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52766/0 (socket says 172.21.15.3:52766) 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e6ffd640 1 -- 172.21.15.3:0/1221052859 learned_addr learned my addr 172.21.15.3:0/1221052859 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951386589 0 0) 0x7f80e809cce0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient(hunting): my global_id is 4466 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80d4003530 con 0x7f80e80a5bd0 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4258930456 0 0) 0x7f80d4003530 con 0x7f80e80a5bd0 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 >> v1:172.21.15.203:6792/0 conn(0x7f80e80a66c0 legacy=0x7f80e813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 >> v1:172.21.15.203:6789/0 conn(0x7f80e80a70c0 legacy=0x7f80e8141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 monclient: found mon.noname-b 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e80742e0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f80e0001bc0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: dump: 2024-05-21T17:21:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.875 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 5 monclient: authenticate success, global_id 4466 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f80e0001de0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 1 -- 172.21.15.3:0/1221052859 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f80e0004f70 con 0x7f80e80a5bd0 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80e5ffb640 10 monclient: dump: 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 4 set_mon_vals no callback set 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 10 monclient: shutdown 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- 172.21.15.3:0/1221052859 >> v1:172.21.15.3:6790/0 conn(0x7f80e80a5bd0 legacy=0x7f80e80a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- 172.21.15.3:0/1221052859 shutdown_connections 2024-05-21T17:21:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- 172.21.15.3:0/1221052859 >> 172.21.15.3:0/1221052859 conn(0x7f80e80ab7d0 msgr2=0x7f80e80a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- 172.21.15.3:0/1221052859 shutdown_connections 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.864+0000 7f80ed965640 1 -- 172.21.15.3:0/1221052859 wait complete. 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: build_initial_monmap 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: monmap: 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:45.879 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 1 Processor -- start 2024-05-21T17:21:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 1 -- start start 2024-05-21T17:21:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: init 2024-05-21T17:21:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: _add_conns ranks=[6,7,3,8,2,1,0,5,4] 2024-05-21T17:21:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient(hunting): picked mon.noname-d con 0x7f80e80a5bd0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient(hunting): picked mon.noname-i con 0x7f80e80a66c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient(hunting): picked mon.noname-g con 0x7f80e80a70c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e809e6a0 con 0x7f80e80a70c0 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e8141970 con 0x7f80e80a5bd0 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e81ce920 con 0x7f80e80a66c0 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient(hunting): authenticate will time out at 973.029213s 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80e6ffd640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f80e80a5bd0 0x7f80e81cdb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:50476/0 (socket says 172.21.15.3:50476) 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80e6ffd640 1 -- 172.21.15.3:0/2927851222 learned_addr learned my addr 172.21.15.3:0/2927851222 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575006802 0 0) 0x7f80e8141970 con 0x7f80e80a5bd0 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): my global_id is 4389 2024-05-21T17:21:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80c8003530 con 0x7f80e80a5bd0 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3337126898 0 0) 0x7f80c8003530 con 0x7f80e80a5bd0 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80e8141970 con 0x7f80e80a5bd0 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f80e0004eb0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): dump: 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557330725 0 0) 0x7f80e809e480 con 0x7f80e80a70c0 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): my global_id is 4539 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80c8003530 con 0x7f80e80a70c0 2024-05-21T17:21:45.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096963283 0 0) 0x7f80e81ce920 con 0x7f80e80a66c0 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): my global_id is 4561 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80e809e480 con 0x7f80e80a66c0 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3734762976 0 0) 0x7f80e8141970 con 0x7f80e80a5bd0 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 >> v1:172.21.15.203:6792/0 conn(0x7f80e80a66c0 legacy=0x7f80e81ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 >> v1:172.21.15.203:6790/0 conn(0x7f80e80a70c0 legacy=0x7f80e81d5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 monclient: found mon.d 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e80a79f0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient: _finish_auth 0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 5 monclient: authenticate success, global_id 4389 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: _renew_subs 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f80e813fbd0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: _renew_subs 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f80e8141f30 con 0x7f80e80a5bd0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f80e0005270 con 0x7f80e80a5bd0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f80e0001da0 con 0x7f80e80a5bd0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80e57fa640 4 set_mon_vals no callback set 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient: dump: 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.888 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f80cc0065d0 con 0x7f80cc002870 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f80e0010740 con 0x7f80e80a5bd0 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7f80e0046220 con 0x7f80e80a5bd0 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80e77fe640 10 monclient: get_auth_request con 0x7f80c8038d90 auth_method 0 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2416554400 0 0) 0x7f80cc0065d0 con 0x7f80cc002870 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 10 monclient(hunting): my global_id is 4532 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.868+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80e8141970 con 0x7f80cc002870 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 506397263 0 0) 0x7f80e8141970 con 0x7f80cc002870 2024-05-21T17:21:45.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80cc0065d0 con 0x7f80cc002870 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f80d00019f0 con 0x7f80cc002870 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 10 monclient: dump: 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4284785281 0 0) 0x7f80cc0065d0 con 0x7f80cc002870 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80cc00a170 con 0x7f80cc002870 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f80d0001cb0 con 0x7f80cc002870 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 >> v1:172.21.15.203:6791/0 conn(0x7f80cc002870 legacy=0x7f80cc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.872+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 >> v1:172.21.15.203:6791/0 conn(0x7f80cc002870 legacy=0x7f80cc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.876+0000 7f80ed965640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.876+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f80cc009e80 con 0x7f80cc009140 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3938658630 0 0) 0x7f80cc009e80 con 0x7f80cc009140 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient(hunting): my global_id is 4541 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80cc0065d0 con 0x7f80cc009140 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2501952242 0 0) 0x7f80cc0065d0 con 0x7f80cc009140 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80cc009e80 con 0x7f80cc009140 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f80d0003b30 con 0x7f80cc009140 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient: dump: 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:45.893 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1390119393 0 0) 0x7f80cc009e80 con 0x7f80cc009140 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 --> v1:172.21.15.203:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f80cc00c420 con 0x7f80cc009140 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 <== mon.5 v1:172.21.15.203:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7705 (unknown 0 0 2774232075) 0x7f80d0005040 con 0x7f80cc009140 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 >> v1:172.21.15.203:6791/0 conn(0x7f80cc009140 legacy=0x7f80cc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80dffff640 1 -- 172.21.15.3:0/2927851222 >> v1:172.21.15.203:6791/0 conn(0x7f80cc009140 legacy=0x7f80cc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:45.895 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 29783, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 19, 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:21:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:45.915 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:45.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 >> v1:172.21.15.203:6812/343827626 conn(0x7f80c8038d90 legacy=0x7f80c803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80ed965640 10 monclient: shutdown 2024-05-21T17:21:45.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80ed965640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:45.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 >> v1:172.21.15.3:6792/0 conn(0x7f80e80a5bd0 legacy=0x7f80e81cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:45.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80e77fe640 1 -- 172.21.15.3:0/2927851222 reap_dead start 2024-05-21T17:21:45.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80e77fe640 1 -- 172.21.15.3:0/2927851222 reap_dead start 2024-05-21T17:21:45.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 shutdown_connections 2024-05-21T17:21:45.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 >> 172.21.15.3:0/2927851222 conn(0x7f80e80ab7d0 msgr2=0x7f80e80a9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:45.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 shutdown_connections 2024-05-21T17:21:45.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:45.880+0000 7f80ed965640 1 -- 172.21.15.3:0/2927851222 wait complete. 2024-05-21T17:21:45.917 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:45.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-05-21T17:21:45.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-05-21T17:21:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient: get_monmap_and_config 2024-05-21T17:21:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient: build_initial_monmap 2024-05-21T17:21:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient: monmap: 2024-05-21T17:21:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:46.033292+0000 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:46.033292+0000 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:46.035 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 1 Processor -- start 2024-05-21T17:21:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 1 -- start start 2024-05-21T17:21:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient: init 2024-05-21T17:21:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient: _add_conns ranks=[5,0,4,8,6,1,7,2,3] 2024-05-21T17:21:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient(hunting): picked mon.noname-h con 0x7ff1bc0a6030 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient(hunting): picked mon.noname-a con 0x7ff1bc0a2b10 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient(hunting): picked mon.noname-c con 0x7ff1bc13f300 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc06cb30 con 0x7ff1bc0a2b10 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc06cfb0 con 0x7ff1bc13f300 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc0a11d0 con 0x7ff1bc0a6030 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient(hunting): authenticate will time out at 973.190904s 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1bb7fe640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7ff1bc13f300 0x7ff1bc1416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:38624/0 (socket says 172.21.15.3:38624) 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1bb7fe640 1 -- 172.21.15.3:0/645920446 learned_addr learned my addr 172.21.15.3:0/645920446 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1ba7fc640 1 --1- 172.21.15.3:0/645920446 >> v1:172.21.15.3:6789/0 conn(0x7ff1bc0a2b10 0x7ff1bc13dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47712/0 (socket says 172.21.15.3:47712) 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894118627 0 0) 0x7ff1bc06cfb0 con 0x7ff1bc13f300 2024-05-21T17:21:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient(hunting): my global_id is 4495 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1a4003530 con 0x7ff1bc13f300 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4005068707 0 0) 0x7ff1bc06cb30 con 0x7ff1bc0a2b10 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient(hunting): my global_id is 4491 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1bc06cfb0 con 0x7ff1bc0a2b10 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3485119934 0 0) 0x7ff1a4003530 con 0x7ff1bc13f300 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 >> v1:172.21.15.203:6791/0 conn(0x7ff1bc0a6030 legacy=0x7ff1bc0a2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 >> v1:172.21.15.3:6789/0 conn(0x7ff1bc0a2b10 legacy=0x7ff1bc13dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 monclient: found mon.noname-c 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:21:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc06cd30 con 0x7ff1bc13f300 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: _finish_auth 0 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 5 monclient: authenticate success, global_id 4495 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff1b4001bc0 con 0x7ff1bc13f300 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:21:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: dump: 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.049 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1b4001d60 con 0x7ff1bc13f300 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:46.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1c1881640 4 set_mon_vals no callback set 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 1 -- 172.21.15.3:0/645920446 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff1b4004f70 con 0x7ff1bc13f300 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.028+0000 7ff1b9ffb640 10 monclient: dump: 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.051 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: shutdown 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- 172.21.15.3:0/645920446 >> v1:172.21.15.3:6791/0 conn(0x7ff1bc13f300 legacy=0x7ff1bc1416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- 172.21.15.3:0/645920446 shutdown_connections 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- 172.21.15.3:0/645920446 >> 172.21.15.3:0/645920446 conn(0x7ff1bc0a93f0 msgr2=0x7ff1bc0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- 172.21.15.3:0/645920446 shutdown_connections 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- 172.21.15.3:0/645920446 wait complete. 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: build_initial_monmap 2024-05-21T17:21:46.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: monmap: 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 Processor -- start 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- start start 2024-05-21T17:21:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: init 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: _add_conns ranks=[7,0,8,1,6,5,4,3,2] 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient(hunting): picked mon.noname-i con 0x7ff1bc0a2b10 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient(hunting): picked mon.noname-a con 0x7ff1bc0a6030 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient(hunting): picked mon.noname-e con 0x7ff1bc13f300 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc13bc50 con 0x7ff1bc0a6030 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc072c50 con 0x7ff1bc0a2b10 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc073350 con 0x7ff1bc13f300 2024-05-21T17:21:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient(hunting): authenticate will time out at 973.194114s 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1ba7fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff1bc0a6030 0x7ff1bc1d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47722/0 (socket says 172.21.15.3:47722) 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1ba7fc640 1 -- 172.21.15.3:0/1736350258 learned_addr learned my addr 172.21.15.3:0/1736350258 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4242491106 0 0) 0x7ff1bc13bc50 con 0x7ff1bc0a6030 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 4500 2024-05-21T17:21:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1a8003530 con 0x7ff1bc0a6030 2024-05-21T17:21:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070331258 0 0) 0x7ff1bc073350 con 0x7ff1bc13f300 2024-05-21T17:21:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 4481 2024-05-21T17:21:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1bc13bc50 con 0x7ff1bc13f300 2024-05-21T17:21:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2075204118 0 0) 0x7ff1a8003530 con 0x7ff1bc0a6030 2024-05-21T17:21:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1bc073350 con 0x7ff1bc0a6030 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff1b0001a80 con 0x7ff1bc0a6030 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): dump: 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.060 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2686914693 0 0) 0x7ff1bc13bc50 con 0x7ff1bc13f300 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1a8003530 con 0x7ff1bc13f300 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff1b4004eb0 con 0x7ff1bc13f300 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:21:46.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): dump: 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.062 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760385916 0 0) 0x7ff1bc072c50 con 0x7ff1bc0a2b10 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 4570 2024-05-21T17:21:46.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1bc13bc50 con 0x7ff1bc0a2b10 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 469451088 0 0) 0x7ff1bc073350 con 0x7ff1bc0a6030 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 >> v1:172.21.15.3:6793/0 conn(0x7ff1bc13f300 legacy=0x7ff1bc1d5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 >> v1:172.21.15.203:6792/0 conn(0x7ff1bc0a2b10 legacy=0x7ff1bc130f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient: _finish_hunting 0 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 monclient: found mon.a 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc06d260 con 0x7ff1bc0a6030 2024-05-21T17:21:46.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient: _finish_auth 0 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient: _check_auth_tickets 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 5 monclient: authenticate success, global_id 4500 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: _renew_subs 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1bc073f60 con 0x7ff1bc0a6030 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: _renew_subs 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1bc131650 con 0x7ff1bc0a6030 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1b0001d20 con 0x7ff1bc0a6030 2024-05-21T17:21:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff1b0004de0 con 0x7ff1bc0a6030 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1b97fa640 4 set_mon_vals no callback set 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1a3fff640 10 monclient: dump: 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.066 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.067 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.068 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:46.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.032+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff19c0065d0 con 0x7ff19c002870 2024-05-21T17:21:46.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7ff1b000ff60 con 0x7ff1bc0a6030 2024-05-21T17:21:46.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7ff1b0045c30 con 0x7ff1bc0a6030 2024-05-21T17:21:46.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1bb7fe640 10 monclient: get_auth_request con 0x7ff1a80393d0 auth_method 0 2024-05-21T17:21:46.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786117604 0 0) 0x7ff19c0065d0 con 0x7ff19c002870 2024-05-21T17:21:46.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:46.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 4579 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1bc073350 con 0x7ff19c002870 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 266611162 0 0) 0x7ff1bc073350 con 0x7ff19c002870 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff19c0065d0 con 0x7ff19c002870 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff1ac0019f0 con 0x7ff19c002870 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient: dump: 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.071 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2974837822 0 0) 0x7ff19c0065d0 con 0x7ff19c002870 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff19c00a170 con 0x7ff19c002870 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff1ac001ca0 con 0x7ff19c002870 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 >> v1:172.21.15.203:6792/0 conn(0x7ff19c002870 legacy=0x7ff19c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.036+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 >> v1:172.21.15.203:6792/0 conn(0x7ff19c002870 legacy=0x7ff19c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:46.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:21:46.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff19c009e80 con 0x7ff19c009140 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404252540 0 0) 0x7ff19c009e80 con 0x7ff19c009140 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 4588 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff19c0065d0 con 0x7ff19c009140 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2318800309 0 0) 0x7ff19c0065d0 con 0x7ff19c009140 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff19c009e80 con 0x7ff19c009140 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff1ac003b70 con 0x7ff19c009140 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient: dump: 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 394054432 0 0) 0x7ff19c009e80 con 0x7ff19c009140 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 --> v1:172.21.15.203:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff19c00c420 con 0x7ff19c009140 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 <== mon.7 v1:172.21.15.203:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7875 (unknown 0 0 2999209000) 0x7ff1ac0050c0 con 0x7ff19c009140 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:21:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:46.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 >> v1:172.21.15.203:6792/0 conn(0x7ff19c009140 legacy=0x7ff19c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1a3fff640 1 -- 172.21.15.3:0/1736350258 >> v1:172.21.15.203:6792/0 conn(0x7ff19c009140 legacy=0x7ff19c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:21:46.078 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:21:46.078 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:46.078 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:46.078 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 12, 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 29927, 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:21:46.079 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 19, 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:46.080 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:46.081 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:21:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:21:46.084 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:21:46.085 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.086 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:21:46.087 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:21:46.087 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.087 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.087 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.087 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:21:46.089 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.093 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.094 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.095 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:21:46.096 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.097 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:21:46.098 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:21:46.099 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.101 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:21:46.102 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:21:46.103 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:46.104 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:46.105 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": [ 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:21:46.106 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:21:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 >> v1:172.21.15.203:6812/343827626 conn(0x7ff1a80393d0 legacy=0x7ff1a803b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 10 monclient: shutdown 2024-05-21T17:21:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 >> v1:172.21.15.3:6789/0 conn(0x7ff1bc0a6030 legacy=0x7ff1bc1d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1bb7fe640 1 -- 172.21.15.3:0/1736350258 reap_dead start 2024-05-21T17:21:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1bb7fe640 1 -- 172.21.15.3:0/1736350258 reap_dead start 2024-05-21T17:21:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 shutdown_connections 2024-05-21T17:21:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 >> 172.21.15.3:0/1736350258 conn(0x7ff1bc0a93f0 msgr2=0x7ff1bc0a36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 shutdown_connections 2024-05-21T17:21:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.044+0000 7ff1c1881640 1 -- 172.21.15.3:0/1736350258 wait complete. 2024-05-21T17:21:46.109 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:21:46.109 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b', 'f'] 2024-05-21T17:21:46.109 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-21T17:21:46.110 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-05-21T17:21:46.110 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-21T17:21:46.110 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-21T17:21:46.110 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:21:46.122 INFO:tasks.ceph.mon.b:Stopped 2024-05-21T17:21:46.122 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-05-21T17:21:46.122 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-05-21T17:21:46.122 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-05-21T17:21:46.122 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:21:46.141 INFO:tasks.ceph.mon.f:Stopped 2024-05-21T17:21:46.141 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-05-21T17:21:46.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-21T17:21:46.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.276+0000 7faa88360640 10 monclient: get_monmap_and_config 2024-05-21T17:21:46.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.276+0000 7faa88360640 10 monclient: build_initial_monmap 2024-05-21T17:21:46.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.276+0000 7faa88360640 10 monclient: monmap: 2024-05-21T17:21:46.283 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:46.283 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.283 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:46.283292+0000 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:46.283292+0000 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:46.284 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 Processor -- start 2024-05-21T17:21:46.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- start start 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient: init 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient: _add_conns ranks=[0,1,7,6,5,3,8,4,2] 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient(hunting): picked mon.noname-a con 0x7faa800a5bd0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient(hunting): picked mon.noname-f con 0x7faa800a66c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient(hunting): picked mon.noname-i con 0x7faa800a70c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8009e480 con 0x7faa800a5bd0 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8009cce0 con 0x7faa800a66c0 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8009cfc0 con 0x7faa800a70c0 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient(hunting): authenticate will time out at 973.441141s 2024-05-21T17:21:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa860fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faa800a5bd0 0x7faa800a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47730/0 (socket says 172.21.15.3:47730) 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa860fc640 1 -- 172.21.15.3:0/4123076076 learned_addr learned my addr 172.21.15.3:0/4123076076 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa858fb640 1 -- 172.21.15.3:0/4123076076 >> v1:172.21.15.203:6789/0 conn(0x7faa800a66c0 legacy=0x7faa8013dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184721576 0 0) 0x7faa8009e480 con 0x7faa800a5bd0 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient(hunting): _negotiate 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient(hunting): my global_id is 4509 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa6c003530 con 0x7faa800a5bd0 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2111180378 0 0) 0x7faa6c003530 con 0x7faa800a5bd0 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 >> v1:172.21.15.203:6792/0 conn(0x7faa800a70c0 legacy=0x7faa80141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 >> v1:172.21.15.203:6789/0 conn(0x7faa800a66c0 legacy=0x7faa8013dc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:21:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: _finish_hunting 0 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 monclient: found mon.noname-a 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa800742e0 con 0x7faa800a5bd0 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: _finish_auth 0 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: _check_auth_tickets 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7faa7c001840 con 0x7faa800a5bd0 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: dump: 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.291 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.292 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 5 monclient: authenticate success, global_id 4509 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa7c001a60 con 0x7faa800a5bd0 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient: get_monmap_and_config success 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 4 set_mon_vals no callback set 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 1 -- 172.21.15.3:0/4123076076 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7faa7c001d50 con 0x7faa800a5bd0 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa850fa640 10 monclient: dump: 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:46.294 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient: shutdown 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- 172.21.15.3:0/4123076076 >> v1:172.21.15.3:6789/0 conn(0x7faa800a5bd0 legacy=0x7faa800a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- 172.21.15.3:0/4123076076 shutdown_connections 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- 172.21.15.3:0/4123076076 >> 172.21.15.3:0/4123076076 conn(0x7faa800ab7d0 msgr2=0x7faa800a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- 172.21.15.3:0/4123076076 shutdown_connections 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- 172.21.15.3:0/4123076076 wait complete. 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient: build_initial_monmap 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient: monmap: 2024-05-21T17:21:46.295 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 Processor -- start 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 1 -- start start 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.280+0000 7faa88360640 10 monclient: init 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 10 monclient: _add_conns ranks=[6,3,2,0,1,4,7,5,8] 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 10 monclient(hunting): picked mon.noname-d con 0x7faa800a5bd0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 10 monclient(hunting): picked mon.noname-g con 0x7faa800a66c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 10 monclient(hunting): picked mon.noname-b con 0x7faa800a70c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8013c680 con 0x7faa800a70c0 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa80142850 con 0x7faa800a66c0 2024-05-21T17:21:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa868fd640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7faa800a70c0 legacy=0x7faa801d5c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:21:46.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa800a79f0 con 0x7faa800a5bd0 2024-05-21T17:21:46.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:46.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa88360640 10 monclient(hunting): authenticate will time out at 973.444282s 2024-05-21T17:21:46.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa860fc640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7faa800a5bd0 0x7faa8013bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:50492/0 (socket says 172.21.15.3:50492) 2024-05-21T17:21:46.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.284+0000 7faa860fc640 1 -- 172.21.15.3:0/1757053986 learned_addr learned my addr 172.21.15.3:0/1757053986 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:46.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.488+0000 7faa868fd640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.3:6790/0 conn(0x7faa800a70c0 legacy=0x7faa801d5c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:21:46.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:46.892+0000 7faa868fd640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.3:6790/0 conn(0x7faa800a70c0 legacy=0x7faa801d5c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:21:47.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:47.696+0000 7faa868fd640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.3:6790/0 conn(0x7faa800a70c0 legacy=0x7faa801d5c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:21:49.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 10 monclient(hunting): tick 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 1 monclient(hunting): continuing hunt 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.3:6792/0 conn(0x7faa800a5bd0 legacy=0x7faa8013bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.203:6790/0 conn(0x7faa800a66c0 legacy=0x7faa801d2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.3:6790/0 conn(0x7faa800a70c0 legacy=0x7faa801d5c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 10 monclient: _add_conns ranks=[7,8,0,1,5,4] 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 10 monclient(hunting): picked mon.noname-i con 0x7faa680037d0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 10 monclient(hunting): picked mon.noname-e con 0x7faa68007460 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 10 monclient(hunting): picked mon.noname-a con 0x7faa6800b1b0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa80123030 con 0x7faa6800b1b0 2024-05-21T17:21:49.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8013c680 con 0x7faa680037d0 2024-05-21T17:21:49.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa68011020 con 0x7faa68007460 2024-05-21T17:21:49.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa75ffb640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316650588 0 0) 0x7faa80123030 con 0x7faa6800b1b0 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient(hunting): my global_id is 4518 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa64003530 con 0x7faa6800b1b0 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2835727082 0 0) 0x7faa64003530 con 0x7faa6800b1b0 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa80123030 con 0x7faa6800b1b0 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7faa78001ed0 con 0x7faa6800b1b0 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:21:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient(hunting): dump: 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1971784219 0 0) 0x7faa80123030 con 0x7faa6800b1b0 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.3:6793/0 conn(0x7faa68007460 legacy=0x7faa680098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.203:6792/0 conn(0x7faa680037d0 legacy=0x7faa68005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 monclient: found mon.a 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:21:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa868fd640 1 -- 172.21.15.3:0/1757053986 reap_dead start 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa68012320 con 0x7faa6800b1b0 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient: _finish_auth 0 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa88360640 5 monclient: authenticate success, global_id 4518 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa88360640 10 monclient: _renew_subs 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa88360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa80123310 con 0x7faa6800b1b0 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa88360640 10 monclient: _renew_subs 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa88360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faa801cd180 con 0x7faa6800b1b0 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa78003c50 con 0x7faa6800b1b0 2024-05-21T17:21:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7faa780041c0 con 0x7faa6800b1b0 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa76ffd640 10 monclient: dump: 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:21:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:21:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:21:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:21:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:21:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:21:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:21:49.297 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.284+0000 7faa848f9640 4 set_mon_vals no callback set 2024-05-21T17:21:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.288+0000 7faa88360640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.288+0000 7faa88360640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.288+0000 7faa88360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.288+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7faa78004360 con 0x7faa6800b1b0 2024-05-21T17:21:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.288+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8006bf50 con 0x7faa6800b1b0 2024-05-21T17:21:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.288+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7faa78045f80 con 0x7faa6800b1b0 2024-05-21T17:21:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.288+0000 7faa860fc640 10 monclient: get_auth_request con 0x7faa64038bb0 auth_method 0 2024-05-21T17:21:49.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.292+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7faa78015f20 con 0x7faa6800b1b0 2024-05-21T17:21:49.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.292+0000 7faa76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:21:49.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.292+0000 7faa76ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:21:49.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.516+0000 7faa88360640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-05-21T17:21:49.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.516+0000 7faa88360640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-05-21T17:21:49.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.516+0000 7faa88360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:21:49.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.516+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faa801cd510 con 0x7faa6800b1b0 2024-05-21T17:21:49.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa76ffd640 1 -- 172.21.15.3:0/1757053986 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2715 (unknown 254746657 0 1064916803) 0x7faa78015820 con 0x7faa6800b1b0 2024-05-21T17:21:49.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-05-21T17:21:49.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa76ffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:21:49.525 INFO:teuthology.orchestra.run.smithi003.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","modified":"2024-05-21T17:20:49.979435Z","created":"2024-05-21T17:20:49.979435Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6790","nonce":0}]},"addr":"172.21.15.3:6790/0","public_addr":"172.21.15.3:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6790","nonce":0}]},"addr":"172.21.15.203:6790/0","public_addr":"172.21.15.203:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6791","nonce":0}]},"addr":"172.21.15.3:6791/0","public_addr":"172.21.15.3:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6791","nonce":0}]},"addr":"172.21.15.203:6791/0","public_addr":"172.21.15.203:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6792","nonce":0}]},"addr":"172.21.15.3:6792/0","public_addr":"172.21.15.3:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6792","nonce":0}]},"addr":"172.21.15.203:6792/0","public_addr":"172.21.15.203:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6793","nonce":0}]},"addr":"172.21.15.3:6793/0","public_addr":"172.21.15.3:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-21T17:21:49.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.203:6812/343827626 conn(0x7faa64038bb0 legacy=0x7faa6403b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:49.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa88360640 10 monclient: shutdown 2024-05-21T17:21:49.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa88360640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:21:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 >> v1:172.21.15.3:6789/0 conn(0x7faa6800b1b0 legacy=0x7faa6800d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 shutdown_connections 2024-05-21T17:21:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 >> 172.21.15.3:0/1757053986 conn(0x7faa800ab7d0 msgr2=0x7faa8009ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:21:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 shutdown_connections 2024-05-21T17:21:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:49.520+0000 7faa88360640 1 -- 172.21.15.3:0/1757053986 wait complete. 2024-05-21T17:21:52.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-21T17:21:52.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.668+0000 7f2faf9b3640 10 monclient: get_monmap_and_config 2024-05-21T17:21:52.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.668+0000 7f2faf9b3640 10 monclient: build_initial_monmap 2024-05-21T17:21:52.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.668+0000 7f2faf9b3640 10 monclient: monmap: 2024-05-21T17:21:52.676 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:21:52.676 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:21:52.676 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:21:52.675945+0000 2024-05-21T17:21:52.676 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:21:52.675945+0000 2024-05-21T17:21:52.676 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:21:52.676 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:21:52.677 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 1 Processor -- start 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 1 -- start start 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 10 monclient: init 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 10 monclient: _reopen_session rank -1 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 10 monclient: _add_conns ranks=[2,3,4,0,5,6,1,8,7] 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 10 monclient(hunting): picked mon.noname-b con 0x7f2fa81347d0 addr v1:172.21.15.3:6790/0 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 10 monclient(hunting): picked mon.noname-g con 0x7f2fa8134bb0 addr v1:172.21.15.203:6790/0 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 10 monclient(hunting): picked mon.noname-c con 0x7f2fa8134fb0 addr v1:172.21.15.3:6791/0 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa809f760 con 0x7f2fa81347d0 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa809fa40 con 0x7f2fa8134bb0 2024-05-21T17:21:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2fad74f640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f2fa81347d0 legacy=0x7f2fa80a09e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:21:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa809fd20 con 0x7f2fa8134fb0 2024-05-21T17:21:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2faf9b3640 10 monclient(hunting): authenticate will time out at 979.833591s 2024-05-21T17:21:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2fadf50640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f2fa8134fb0 0x7f2fa813df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:38646/0 (socket says 172.21.15.3:38646) 2024-05-21T17:21:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.672+0000 7f2fadf50640 1 -- 172.21.15.3:0/2617135675 learned_addr learned my addr 172.21.15.3:0/2617135675 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:21:52.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:52.876+0000 7f2fad74f640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.3:6790/0 conn(0x7f2fa81347d0 legacy=0x7f2fa80a09e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:21:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:53.280+0000 7f2fad74f640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.3:6790/0 conn(0x7f2fa81347d0 legacy=0x7f2fa80a09e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:21:54.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:54.084+0000 7f2fad74f640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.3:6790/0 conn(0x7f2fa81347d0 legacy=0x7f2fa80a09e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:21:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 10 monclient(hunting): tick 2024-05-21T17:21:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 10 monclient(hunting): _check_auth_tickets 2024-05-21T17:21:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 1 monclient(hunting): continuing hunt 2024-05-21T17:21:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 10 monclient(hunting): _reopen_session rank -1 2024-05-21T17:21:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.3:6791/0 conn(0x7f2fa8134fb0 legacy=0x7f2fa813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.203:6790/0 conn(0x7f2fa8134bb0 legacy=0x7f2fa813a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.3:6790/0 conn(0x7f2fa81347d0 legacy=0x7f2fa80a09e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:21:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 10 monclient: _add_conns ranks=[0,6,5,8,7,1] 2024-05-21T17:21:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c0037d0 addr v1:172.21.15.3:6789/0 2024-05-21T17:21:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 10 monclient(hunting): picked mon.noname-d con 0x7f2f8c0073c0 addr v1:172.21.15.3:6792/0 2024-05-21T17:21:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 10 monclient(hunting): picked mon.noname-h con 0x7f2f8c00b150 addr v1:172.21.15.203:6791/0 2024-05-21T17:21:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa80a7930 con 0x7f2f8c0037d0 2024-05-21T17:21:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa809f760 con 0x7f2f8c00b150 2024-05-21T17:21:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f8c011010 con 0x7f2f8c0073c0 2024-05-21T17:21:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:55.672+0000 7f2f9effd640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:58.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 10 monclient(hunting): tick 2024-05-21T17:21:58.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 10 monclient(hunting): _check_auth_tickets 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 1 monclient(hunting): continuing hunt 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 10 monclient(hunting): _reopen_session rank -1 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.3:6792/0 conn(0x7f2f8c0073c0 legacy=0x7f2f8c009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.203:6791/0 conn(0x7f2f8c00b150 legacy=0x7f2f8c00d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.3:6789/0 conn(0x7f2f8c0037d0 legacy=0x7f2f8c005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 10 monclient: _add_conns ranks=[8,1,7] 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 10 monclient(hunting): picked mon.noname-e con 0x7f2f8c00f090 addr v1:172.21.15.3:6793/0 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2fadf50640 1 -- 172.21.15.3:0/2617135675 reap_dead start 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 10 monclient(hunting): picked mon.noname-f con 0x7f2f8c0111e0 addr v1:172.21.15.203:6789/0 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 10 monclient(hunting): picked mon.noname-i con 0x7f2f8c011dd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f8c0122b0 con 0x7f2f8c0111e0 2024-05-21T17:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f8c006350 con 0x7f2f8c011dd0 2024-05-21T17:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 1 -- 172.21.15.3:0/2617135675 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f8c006570 con 0x7f2f8c00f090 2024-05-21T17:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2f9effd640 10 monclient(hunting): _renew_subs 2024-05-21T17:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2fadf50640 1 -- 172.21.15.3:0/2617135675 reap_dead start 2024-05-21T17:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.676+0000 7f2facf4e640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.203:6789/0 conn(0x7f2f8c0111e0 legacy=0x7f2f8c011690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:21:58.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:58.880+0000 7f2facf4e640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.203:6789/0 conn(0x7f2f8c0111e0 legacy=0x7f2f8c011690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:21:59.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:21:59.284+0000 7f2facf4e640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.203:6789/0 conn(0x7f2f8c0111e0 legacy=0x7f2f8c011690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:22:00.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.088+0000 7f2facf4e640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.203:6789/0 conn(0x7f2f8c0111e0 legacy=0x7f2f8c011690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.304+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147594054 0 0) 0x7f2f8c006570 con 0x7f2f8c00f090 2024-05-21T17:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.304+0000 7f2f9ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.304+0000 7f2f9ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.304+0000 7f2f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.304+0000 7f2f9ffff640 10 monclient(hunting): my global_id is 4490 2024-05-21T17:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.304+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f94003530 con 0x7f2f8c00f090 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.304+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3867486640 0 0) 0x7f2f94003530 con 0x7f2f8c00f090 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.203:6792/0 conn(0x7f2f8c011dd0 legacy=0x7f2f8c0196c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.203:6789/0 conn(0x7f2f8c0111e0 legacy=0x7f2f8c011690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: _finish_hunting 0 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 1 monclient: found mon.noname-e 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8c005c70 con 0x7f2f8c00f090 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: _finish_auth 0 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: _check_auth_tickets 2024-05-21T17:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2fa4004340 con 0x7f2f8c00f090 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: dump: 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 5 monclient: authenticate success, global_id 4490 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2fa40044e0 con 0x7f2f8c00f090 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 10 monclient: get_monmap_and_config success 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 1 -- 172.21.15.3:0/2617135675 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2fa4004b10 con 0x7f2f8c00f090 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 4 set_mon_vals no callback set 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2f9ffff640 10 monclient: dump: 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.317 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 10 monclient: shutdown 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 1 -- 172.21.15.3:0/2617135675 >> v1:172.21.15.3:6793/0 conn(0x7f2f8c00f090 legacy=0x7f2f8c00f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 1 -- 172.21.15.3:0/2617135675 shutdown_connections 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 1 -- 172.21.15.3:0/2617135675 >> 172.21.15.3:0/2617135675 conn(0x7f2fa80a72c0 msgr2=0x7f2fa80a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 1 -- 172.21.15.3:0/2617135675 shutdown_connections 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 1 -- 172.21.15.3:0/2617135675 wait complete. 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 10 monclient: build_initial_monmap 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 10 monclient: monmap: 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:00.320 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 1 Processor -- start 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 1 -- start start 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.308+0000 7f2faf9b3640 10 monclient: init 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 10 monclient: _add_conns ranks=[1,5,4,2,7,3,6,0,8] 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 10 monclient(hunting): picked mon.noname-f con 0x7f2f8c00f090 addr v1:172.21.15.203:6789/0 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 10 monclient(hunting): picked mon.noname-h con 0x7f2f8c0111e0 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8c011dd0 addr v1:172.21.15.3:6791/0 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa8125ba0 con 0x7f2f8c00f090 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa8127050 con 0x7f2f8c011dd0 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa8073b30 con 0x7f2f8c0111e0 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2faf9b3640 10 monclient(hunting): authenticate will time out at 987.472208s 2024-05-21T17:22:00.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2fad74f640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f2f8c00f090 legacy=0x7f2fa8132900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:22:00.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2fadf50640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f2f8c011dd0 0x7f2fa81d60f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:38404/0 (socket says 172.21.15.3:38404) 2024-05-21T17:22:00.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.312+0000 7f2fadf50640 1 -- 172.21.15.3:0/3012616412 learned_addr learned my addr 172.21.15.3:0/3012616412 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:00.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 365600393 0 0) 0x7f2fa8127050 con 0x7f2f8c011dd0 2024-05-21T17:22:00.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient(hunting): my global_id is 4504 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f8c0121b0 con 0x7f2f8c011dd0 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1835377178 0 0) 0x7f2f8c0121b0 con 0x7f2f8c011dd0 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2fa8127050 con 0x7f2f8c011dd0 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2fa0001c10 con 0x7f2f8c011dd0 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient(hunting): dump: 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.338 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.339 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1144941254 0 0) 0x7f2fa8127050 con 0x7f2f8c011dd0 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 >> v1:172.21.15.203:6791/0 conn(0x7f2f8c0111e0 legacy=0x7f2fa81d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 >> v1:172.21.15.203:6789/0 conn(0x7f2f8c00f090 legacy=0x7f2fa8132900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient: _finish_hunting 0 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 monclient: found mon.c 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa80a31c0 con 0x7f2f8c011dd0 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient: _finish_auth 0 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient: _check_auth_tickets 2024-05-21T17:22:00.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 5 monclient: authenticate success, global_id 4504 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 10 monclient: _renew_subs 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa8126d70 con 0x7f2f8c011dd0 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 10 monclient: _renew_subs 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f2fa8133010 con 0x7f2f8c011dd0 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2fa00020c0 con 0x7f2f8c011dd0 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2fa0004ef0 con 0x7f2f8c011dd0 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9f7fe640 4 set_mon_vals no callback set 2024-05-21T17:22:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 10 monclient: dump: 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.342 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.343 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.343 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.343 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.343 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.343 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f2fa0005090 con 0x7f2f8c011dd0 2024-05-21T17:22:00.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:00.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:00.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:22:00.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7f2fa0045e20 con 0x7f2f8c011dd0 2024-05-21T17:22:00.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f88005090 con 0x7f2f8c011dd0 2024-05-21T17:22:00.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.332+0000 7f2fad74f640 10 monclient: get_auth_request con 0x7f2f8c03b170 auth_method 0 2024-05-21T17:22:00.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.340+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f2fa0015d60 con 0x7f2f8c011dd0 2024-05-21T17:22:00.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.340+0000 7f2f9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:00.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.340+0000 7f2f9dffb640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:22:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.588+0000 7f2faf9b3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-05-21T17:22:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.588+0000 7f2faf9b3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-05-21T17:22:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.588+0000 7f2faf9b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:22:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.588+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2f88005380 con 0x7f2f8c011dd0 2024-05-21T17:22:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.588+0000 7f2f9dffb640 1 -- 172.21.15.3:0/3012616412 <== mon.4 v1:172.21.15.3:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 923634177) 0x7f2fa0015660 con 0x7f2f8c011dd0 2024-05-21T17:22:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.588+0000 7f2f9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-05-21T17:22:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.588+0000 7f2f9dffb640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:22:00.594 INFO:teuthology.orchestra.run.smithi003.stdout:{"election_epoch":16,"quorum":[0,3,4,5,6,7,8],"quorum_names":["a","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","modified":"2024-05-21T17:20:49.979435Z","created":"2024-05-21T17:20:49.979435Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6790","nonce":0}]},"addr":"172.21.15.3:6790/0","public_addr":"172.21.15.3:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6790","nonce":0}]},"addr":"172.21.15.203:6790/0","public_addr":"172.21.15.203:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6791","nonce":0}]},"addr":"172.21.15.3:6791/0","public_addr":"172.21.15.3:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6791","nonce":0}]},"addr":"172.21.15.203:6791/0","public_addr":"172.21.15.203:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6792","nonce":0}]},"addr":"172.21.15.3:6792/0","public_addr":"172.21.15.3:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6792","nonce":0}]},"addr":"172.21.15.203:6792/0","public_addr":"172.21.15.203:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6793","nonce":0}]},"addr":"172.21.15.3:6793/0","public_addr":"172.21.15.3:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-21T17:22:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.592+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 >> v1:172.21.15.203:6812/343827626 conn(0x7f2f8c03b170 legacy=0x7f2f8c03d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.592+0000 7f2faf9b3640 10 monclient: shutdown 2024-05-21T17:22:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.592+0000 7f2faf9b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.592+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 >> v1:172.21.15.3:6791/0 conn(0x7f2f8c011dd0 legacy=0x7f2fa81d60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.592+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 shutdown_connections 2024-05-21T17:22:00.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.592+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 >> 172.21.15.3:0/3012616412 conn(0x7f2fa80a72c0 msgr2=0x7f2fa80a4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:00.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.592+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 shutdown_connections 2024-05-21T17:22:00.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.592+0000 7f2faf9b3640 1 -- 172.21.15.3:0/3012616412 wait complete. 2024-05-21T17:22:00.614 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-05-21T17:22:00.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-05-21T17:22:00.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-21T17:22:00.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient: get_monmap_and_config 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient: build_initial_monmap 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient: monmap: 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:22:00.737000+0000 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:22:00.737000+0000 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:00.738 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 1 Processor -- start 2024-05-21T17:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 1 -- start start 2024-05-21T17:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient: init 2024-05-21T17:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient: _add_conns ranks=[3,8,5,1,0,7,6,4,2] 2024-05-21T17:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient(hunting): picked mon.noname-g con 0x7fc86c135880 addr v1:172.21.15.203:6790/0 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient(hunting): picked mon.noname-e con 0x7fc86c09dcb0 addr v1:172.21.15.3:6793/0 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient(hunting): picked mon.noname-h con 0x7fc86c13f320 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c136370 con 0x7fc86c135880 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c09cff0 con 0x7fc86c13f320 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c0684d0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient(hunting): authenticate will time out at 987.894687s 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc871339640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fc86c09dcb0 0x7fc86c13dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:38428/0 (socket says 172.21.15.3:38428) 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc871339640 1 -- 172.21.15.3:0/1862999968 learned_addr learned my addr 172.21.15.3:0/1862999968 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4211066425 0 0) 0x7fc86c0684d0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient(hunting): my global_id is 4499 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc858003530 con 0x7fc86c09dcb0 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1547881002 0 0) 0x7fc858003530 con 0x7fc86c09dcb0 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 >> v1:172.21.15.203:6791/0 conn(0x7fc86c13f320 legacy=0x7fc86c141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 >> v1:172.21.15.203:6790/0 conn(0x7fc86c135880 legacy=0x7fc86c135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: _finish_hunting 0 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 monclient: found mon.noname-e 2024-05-21T17:22:00.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c09ca30 con 0x7fc86c09dcb0 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: _finish_auth 0 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: _check_auth_tickets 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc85c001840 con 0x7fc86c09dcb0 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 5 monclient: authenticate success, global_id 4499 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: dump: 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc85c0019e0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 10 monclient: get_monmap_and_config success 2024-05-21T17:22:00.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc873d9e640 4 set_mon_vals no callback set 2024-05-21T17:22:00.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 1 -- 172.21.15.3:0/1862999968 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc85c004bf0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:22:00.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.732+0000 7fc870b38640 10 monclient: dump: 2024-05-21T17:22:00.752 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.752 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.753 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: shutdown 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- 172.21.15.3:0/1862999968 >> v1:172.21.15.3:6793/0 conn(0x7fc86c09dcb0 legacy=0x7fc86c13dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- 172.21.15.3:0/1862999968 shutdown_connections 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- 172.21.15.3:0/1862999968 >> 172.21.15.3:0/1862999968 conn(0x7fc86c0a8370 msgr2=0x7fc86c0a4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- 172.21.15.3:0/1862999968 shutdown_connections 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- 172.21.15.3:0/1862999968 wait complete. 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: build_initial_monmap 2024-05-21T17:22:00.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: monmap: 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 Processor -- start 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- start start 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: init 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: _add_conns ranks=[8,3,2,7,5,6,1,4,0] 2024-05-21T17:22:00.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient(hunting): picked mon.noname-e con 0x7fc86c09dcb0 addr v1:172.21.15.3:6793/0 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient(hunting): picked mon.noname-g con 0x7fc86c135880 addr v1:172.21.15.203:6790/0 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient(hunting): picked mon.noname-b con 0x7fc86c13f320 addr v1:172.21.15.3:6790/0 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c13ab40 con 0x7fc86c13f320 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c13a7f0 con 0x7fc86c135880 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c09eae0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient(hunting): authenticate will time out at 987.897874s 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc87233b640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7fc86c13f320 legacy=0x7fc86c1d5f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc871b3a640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fc86c09dcb0 0x7fc86c1399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:38434/0 (socket says 172.21.15.3:38434) 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc871b3a640 1 -- 172.21.15.3:0/2774441463 learned_addr learned my addr 172.21.15.3:0/2774441463 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 891812876 0 0) 0x7fc86c09eae0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient(hunting): my global_id is 4508 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc850003530 con 0x7fc86c09dcb0 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1711098029 0 0) 0x7fc850003530 con 0x7fc86c09dcb0 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc86c09eae0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc868001b80 con 0x7fc86c09dcb0 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient(hunting): dump: 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.759 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3336238227 0 0) 0x7fc86c09eae0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 >> v1:172.21.15.203:6790/0 conn(0x7fc86c135880 legacy=0x7fc86c13a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 >> v1:172.21.15.3:6790/0 conn(0x7fc86c13f320 legacy=0x7fc86c1d5f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient: _finish_hunting 0 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 monclient: found mon.e 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c09e690 con 0x7fc86c09dcb0 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient: _finish_auth 0 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient: _check_auth_tickets 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 5 monclient: authenticate success, global_id 4508 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: _renew_subs 2024-05-21T17:22:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc86c13ce30 con 0x7fc86c09dcb0 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: _renew_subs 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc86c13e6e0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8680038d0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc868003f00 con 0x7fc86c09dcb0 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc863fff640 4 set_mon_vals no callback set 2024-05-21T17:22:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 10 monclient: dump: 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc8540065d0 con 0x7fc854002870 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.736+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fc868004120 con 0x7fc86c09dcb0 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7fc868045eb0 con 0x7fc86c09dcb0 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 843856647 0 0) 0x7fc8540065d0 con 0x7fc854002870 2024-05-21T17:22:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient(hunting): my global_id is 4527 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc86c09eae0 con 0x7fc854002870 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2963742449 0 0) 0x7fc86c09eae0 con 0x7fc854002870 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8540065d0 con 0x7fc854002870 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc85c001800 con 0x7fc854002870 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient: dump: 2024-05-21T17:22:00.765 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.766 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc87233b640 10 monclient: get_auth_request con 0x7fc850038b00 auth_method 0 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2786802124 0 0) 0x7fc8540065d0 con 0x7fc854002870 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc85400a170 con 0x7fc854002870 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc85c004aa0 con 0x7fc854002870 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:22:00.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 >> v1:172.21.15.3:6789/0 conn(0x7fc854002870 legacy=0x7fc854004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.740+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 >> v1:172.21.15.3:6789/0 conn(0x7fc854002870 legacy=0x7fc854004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:00.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc873d9e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:22:00.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc854009e80 con 0x7fc854009140 2024-05-21T17:22:00.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347418 0 0) 0x7fc854009e80 con 0x7fc854009140 2024-05-21T17:22:00.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 10 monclient(hunting): my global_id is 4536 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8540065d0 con 0x7fc854009140 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2730182602 0 0) 0x7fc8540065d0 con 0x7fc854009140 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc854009e80 con 0x7fc854009140 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc85c001cd0 con 0x7fc854009140 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:22:00.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 10 monclient: dump: 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.771 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2201996436 0 0) 0x7fc854009e80 con 0x7fc854009140 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.744+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 --> v1:172.21.15.3:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc85400c420 con 0x7fc854009140 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 <== mon.0 v1:172.21.15.3:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 2023707981) 0x7fc85c001d40 con 0x7fc854009140 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc8627fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc8627fc640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 >> v1:172.21.15.3:6789/0 conn(0x7fc854009140 legacy=0x7fc8540095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc8627fc640 1 -- 172.21.15.3:0/2774441463 >> v1:172.21.15.3:6789/0 conn(0x7fc854009140 legacy=0x7fc8540095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:00.773 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:22:00.773 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:22:00.773 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:22:00.773 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "leader", 2024-05-21T17:22:00.773 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 16, 2024-05-21T17:22:00.773 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 44679, 2024-05-21T17:22:00.773 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:22:00.773 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 0, 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:00.774 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:00.775 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:22:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:22:00.778 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:00.779 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:22:00.780 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.781 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.783 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:22:00.784 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.787 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.789 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:22:00.790 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.791 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:22:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:22:00.795 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:22:00.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.797 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:22:00.798 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:22:00.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 >> v1:172.21.15.203:6812/343827626 conn(0x7fc850038b00 legacy=0x7fc85003afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc873d9e640 10 monclient: shutdown 2024-05-21T17:22:00.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc873d9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:00.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 >> v1:172.21.15.3:6793/0 conn(0x7fc86c09dcb0 legacy=0x7fc86c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc87233b640 1 -- 172.21.15.3:0/2774441463 reap_dead start 2024-05-21T17:22:00.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc87233b640 1 -- 172.21.15.3:0/2774441463 reap_dead start 2024-05-21T17:22:00.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 shutdown_connections 2024-05-21T17:22:00.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 >> 172.21.15.3:0/2774441463 conn(0x7fc86c0a8370 msgr2=0x7fc86c141c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:00.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 shutdown_connections 2024-05-21T17:22:00.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.748+0000 7fc873d9e640 1 -- 172.21.15.3:0/2774441463 wait complete. 2024-05-21T17:22:00.800 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-05-21T17:22:00.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-05-21T17:22:00.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-21T17:22:00.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient: get_monmap_and_config 2024-05-21T17:22:00.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient: build_initial_monmap 2024-05-21T17:22:00.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient: monmap: 2024-05-21T17:22:00.917 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:00.917 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.917 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:22:00.917182+0000 2024-05-21T17:22:00.917 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:22:00.917182+0000 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:00.919 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 1 Processor -- start 2024-05-21T17:22:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 1 -- start start 2024-05-21T17:22:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient: init 2024-05-21T17:22:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient: _add_conns ranks=[8,5,2,7,4,0,6,1,3] 2024-05-21T17:22:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient(hunting): picked mon.noname-e con 0x7fc94c0734b0 addr v1:172.21.15.3:6793/0 2024-05-21T17:22:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient(hunting): picked mon.noname-h con 0x7fc94c0a30d0 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient(hunting): picked mon.noname-b con 0x7fc94c070d10 addr v1:172.21.15.3:6790/0 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc94c073fa0 con 0x7fc94c070d10 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc94c0a1e00 con 0x7fc94c0a30d0 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc94c0a20e0 con 0x7fc94c0734b0 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient(hunting): authenticate will time out at 988.074816s 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94bfff640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7fc94c070d10 legacy=0x7fc94c13f000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94b7fe640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fc94c0734b0 0x7fc94c073890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:38436/0 (socket says 172.21.15.3:38436) 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94b7fe640 1 -- 172.21.15.3:0/612047712 learned_addr learned my addr 172.21.15.3:0/612047712 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54510069 0 0) 0x7fc94c0a20e0 con 0x7fc94c0734b0 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient(hunting): my global_id is 4517 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc938003530 con 0x7fc94c0734b0 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3471266025 0 0) 0x7fc938003530 con 0x7fc94c0734b0 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 >> v1:172.21.15.203:6791/0 conn(0x7fc94c0a30d0 legacy=0x7fc94c13b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 >> v1:172.21.15.3:6790/0 conn(0x7fc94c070d10 legacy=0x7fc94c13f000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 monclient: found mon.noname-e 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:22:00.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c0a23c0 con 0x7fc94c0734b0 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: _finish_auth 0 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc934001840 con 0x7fc94c0734b0 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 5 monclient: authenticate success, global_id 4517 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: dump: 2024-05-21T17:22:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.930 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc934001a60 con 0x7fc94c0734b0 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 1 -- 172.21.15.3:0/612047712 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc934004bf0 con 0x7fc94c0734b0 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:22:00.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc94a7fc640 10 monclient: dump: 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 10 monclient: get_monmap_and_config success 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.912+0000 7fc951f41640 4 set_mon_vals no callback set 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: shutdown 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- 172.21.15.3:0/612047712 >> v1:172.21.15.3:6793/0 conn(0x7fc94c0734b0 legacy=0x7fc94c073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- 172.21.15.3:0/612047712 shutdown_connections 2024-05-21T17:22:00.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- 172.21.15.3:0/612047712 >> 172.21.15.3:0/612047712 conn(0x7fc94c0a93f0 msgr2=0x7fc94c0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- 172.21.15.3:0/612047712 shutdown_connections 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- 172.21.15.3:0/612047712 wait complete. 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: build_initial_monmap 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: monmap: 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:22:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 Processor -- start 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- start start 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: init 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: _add_conns ranks=[8,2,4,6,3,7,1,5,0] 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient(hunting): picked mon.noname-e con 0x7fc94c070d10 addr v1:172.21.15.3:6793/0 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient(hunting): picked mon.noname-b con 0x7fc94c0734b0 addr v1:172.21.15.3:6790/0 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient(hunting): picked mon.noname-c con 0x7fc94c0a30d0 addr v1:172.21.15.3:6791/0 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc94c06d2f0 con 0x7fc94c0734b0 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc94c0a11a0 con 0x7fc94c0a30d0 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc94c06d810 con 0x7fc94c070d10 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient(hunting): authenticate will time out at 988.077820s 2024-05-21T17:22:00.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc94affd640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7fc94c0734b0 legacy=0x7fc94c06cbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc94b7fe640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fc94c070d10 0x7fc94c06c4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:38442/0 (socket says 172.21.15.3:38442) 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc94b7fe640 1 -- 172.21.15.3:0/2342606100 learned_addr learned my addr 172.21.15.3:0/2342606100 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2289862606 0 0) 0x7fc94c06d810 con 0x7fc94c070d10 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): my global_id is 4526 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc92c003530 con 0x7fc94c070d10 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3457328319 0 0) 0x7fc94c0a11a0 con 0x7fc94c0a30d0 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): my global_id is 4513 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc94c06d810 con 0x7fc94c0a30d0 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4210809495 0 0) 0x7fc94c06d810 con 0x7fc94c0a30d0 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc94c0a11a0 con 0x7fc94c0a30d0 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3389623447 0 0) 0x7fc92c003530 con 0x7fc94c070d10 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc94c06d810 con 0x7fc94c070d10 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc944004560 con 0x7fc94c0a30d0 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): dump: 2024-05-21T17:22:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.939 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc934004b30 con 0x7fc94c070d10 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient(hunting): dump: 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.940 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3939491023 0 0) 0x7fc94c0a11a0 con 0x7fc94c0a30d0 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 >> v1:172.21.15.3:6793/0 conn(0x7fc94c070d10 legacy=0x7fc94c06c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 >> v1:172.21.15.3:6790/0 conn(0x7fc94c0734b0 legacy=0x7fc94c06cbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient: _finish_hunting 0 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 monclient: found mon.c 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:22:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c120010 con 0x7fc94c0a30d0 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient: _finish_auth 0 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient: _check_auth_tickets 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 5 monclient: authenticate success, global_id 4513 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: _renew_subs 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc94c06e370 con 0x7fc94c0a30d0 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: _renew_subs 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fc94c140040 con 0x7fc94c0a30d0 2024-05-21T17:22:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9440049b0 con 0x7fc94c0a30d0 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc944004ee0 con 0x7fc94c0a30d0 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc949ffb640 4 set_mon_vals no callback set 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 10 monclient: dump: 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.945 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc9300065d0 con 0x7fc930002870 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fc944005080 con 0x7fc94c0a30d0 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.916+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7fc944045e90 con 0x7fc94c0a30d0 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592693438 0 0) 0x7fc9300065d0 con 0x7fc930002870 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient(hunting): my global_id is 4522 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc94c0a11a0 con 0x7fc930002870 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2263768427 0 0) 0x7fc94c0a11a0 con 0x7fc930002870 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9300065d0 con 0x7fc930002870 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc934001960 con 0x7fc930002870 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient: dump: 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.947 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.948 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc94affd640 10 monclient: get_auth_request con 0x7fc92c0390a0 auth_method 0 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 382069219 0 0) 0x7fc9300065d0 con 0x7fc930002870 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc93000a170 con 0x7fc930002870 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc9340072d0 con 0x7fc930002870 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 >> v1:172.21.15.3:6791/0 conn(0x7fc930002870 legacy=0x7fc930004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.920+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 >> v1:172.21.15.3:6791/0 conn(0x7fc930002870 legacy=0x7fc930004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:00.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc951f41640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:22:00.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc930009e60 con 0x7fc930009120 2024-05-21T17:22:00.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176393571 0 0) 0x7fc930009e60 con 0x7fc930009120 2024-05-21T17:22:00.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:00.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:00.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:00.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 10 monclient(hunting): my global_id is 4531 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9300065d0 con 0x7fc930009120 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3638902943 0 0) 0x7fc9300065d0 con 0x7fc930009120 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc930009e60 con 0x7fc930009120 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc934005980 con 0x7fc930009120 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 10 monclient: dump: 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:00.952 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3799655499 0 0) 0x7fc930009e60 con 0x7fc930009120 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.924+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 --> v1:172.21.15.3:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc93000c400 con 0x7fc930009120 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 <== mon.4 v1:172.21.15.3:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7682 (unknown 0 0 28380497) 0x7fc934006e90 con 0x7fc930009120 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc943fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc943fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 >> v1:172.21.15.3:6791/0 conn(0x7fc930009120 legacy=0x7fc9300095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc943fff640 1 -- 172.21.15.3:0/2342606100 >> v1:172.21.15.3:6791/0 conn(0x7fc930009120 legacy=0x7fc9300095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 16, 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 44859, 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:22:00.955 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 0, 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:22:00.956 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:00.957 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:00.958 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:22:00.959 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:22:00.960 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:00.961 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.964 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.969 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:22:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.974 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.977 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.978 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 5 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:22:00.979 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:22:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 >> v1:172.21.15.203:6812/343827626 conn(0x7fc92c0390a0 legacy=0x7fc92c03b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc951f41640 10 monclient: shutdown 2024-05-21T17:22:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc951f41640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 >> v1:172.21.15.3:6791/0 conn(0x7fc94c0a30d0 legacy=0x7fc94c1d59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc94bfff640 1 -- 172.21.15.3:0/2342606100 reap_dead start 2024-05-21T17:22:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc94bfff640 1 -- 172.21.15.3:0/2342606100 reap_dead start 2024-05-21T17:22:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 shutdown_connections 2024-05-21T17:22:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 >> 172.21.15.3:0/2342606100 conn(0x7fc94c0a93f0 msgr2=0x7fc94c0a4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:00.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 shutdown_connections 2024-05-21T17:22:00.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:00.928+0000 7fc951f41640 1 -- 172.21.15.3:0/2342606100 wait complete. 2024-05-21T17:22:00.982 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-05-21T17:22:00.982 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-05-21T17:22:00.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-05-21T17:22:01.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient: get_monmap_and_config 2024-05-21T17:22:01.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient: monmap: 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:22:01.100757+0000 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:22:01.100757+0000 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.102 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.102 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.102 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 Processor -- start 2024-05-21T17:22:01.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- start start 2024-05-21T17:22:01.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient: init 2024-05-21T17:22:01.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient: _add_conns ranks=[3,6,7,5,4,8,0,1,2] 2024-05-21T17:22:01.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient(hunting): picked mon.noname-g con 0x7fbde00a48c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient(hunting): picked mon.noname-d con 0x7fbde00a4ca0 addr v1:172.21.15.3:6792/0 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient(hunting): picked mon.noname-i con 0x7fbde013f210 addr v1:172.21.15.203:6792/0 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde00a88a0 con 0x7fbde00a48c0 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde00a6550 con 0x7fbde00a4ca0 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde00a28c0 con 0x7fbde013f210 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient(hunting): authenticate will time out at 988.258188s 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdde7fc640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7fbde00a4ca0 0x7fbde013daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:55404/0 (socket says 172.21.15.3:55404) 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdde7fc640 1 -- 172.21.15.3:0/3960055765 learned_addr learned my addr 172.21.15.3:0/3960055765 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1739669268 0 0) 0x7fbde00a6550 con 0x7fbde00a4ca0 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient(hunting): my global_id is 4398 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc8003530 con 0x7fbde00a4ca0 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1073198963 0 0) 0x7fbdc8003530 con 0x7fbde00a4ca0 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 >> v1:172.21.15.203:6792/0 conn(0x7fbde013f210 legacy=0x7fbde0141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 >> v1:172.21.15.203:6790/0 conn(0x7fbde00a48c0 legacy=0x7fbde00a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 monclient: found mon.noname-d 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde00a2ba0 con 0x7fbde00a4ca0 2024-05-21T17:22:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbdd4001840 con 0x7fbde00a4ca0 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: dump: 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 5 monclient: authenticate success, global_id 4398 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdd4001a60 con 0x7fbde00a4ca0 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient: get_monmap_and_config success 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 1 -- 172.21.15.3:0/3960055765 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbdd4001d50 con 0x7fbde00a4ca0 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 4 set_mon_vals no callback set 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbdddffb640 10 monclient: dump: 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 10 monclient: shutdown 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- 172.21.15.3:0/3960055765 >> v1:172.21.15.3:6792/0 conn(0x7fbde00a4ca0 legacy=0x7fbde013daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- 172.21.15.3:0/3960055765 shutdown_connections 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- 172.21.15.3:0/3960055765 >> 172.21.15.3:0/3960055765 conn(0x7fbde00ab5b0 msgr2=0x7fbde00a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- 172.21.15.3:0/3960055765 shutdown_connections 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.096+0000 7fbde58c8640 1 -- 172.21.15.3:0/3960055765 wait complete. 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: monmap: 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 1 Processor -- start 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 1 -- start start 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: init 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: _add_conns ranks=[2,1,5,0,8,7,6,4,3] 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient(hunting): picked mon.noname-b con 0x7fbde00a48c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient(hunting): picked mon.noname-f con 0x7fbde00a4ca0 addr v1:172.21.15.203:6789/0 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient(hunting): picked mon.noname-h con 0x7fbde013f210 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde00a7110 con 0x7fbde00a4ca0 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde009e670 con 0x7fbde00a48c0 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde00a0960 con 0x7fbde013f210 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient(hunting): authenticate will time out at 988.261412s 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbddeffd640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7fbde00a48c0 legacy=0x7fbde00a0d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdde7fc640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fbde00a4ca0 legacy=0x7fbde01d1db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbddf7fe640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7fbde013f210 0x7fbde01d54e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.3:38222/0 (socket says 172.21.15.3:38222) 2024-05-21T17:22:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbddf7fe640 1 -- 172.21.15.3:0/1957356715 learned_addr learned my addr 172.21.15.3:0/1957356715 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280448675 0 0) 0x7fbde00a0960 con 0x7fbde013f210 2024-05-21T17:22:01.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient(hunting): my global_id is 4577 2024-05-21T17:22:01.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdcc003530 con 0x7fbde013f210 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2927213146 0 0) 0x7fbdcc003530 con 0x7fbde013f210 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbde00a0960 con 0x7fbde013f210 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbdd8001e40 con 0x7fbde013f210 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient(hunting): dump: 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3756577311 0 0) 0x7fbde00a0960 con 0x7fbde013f210 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 >> v1:172.21.15.3:6790/0 conn(0x7fbde00a48c0 legacy=0x7fbde00a0d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 >> v1:172.21.15.203:6789/0 conn(0x7fbde00a4ca0 legacy=0x7fbde01d1db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:01.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 monclient: found mon.h 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.203:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde009ef20 con 0x7fbde013f210 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 5 monclient: authenticate success, global_id 4577 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: _renew_subs 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.203:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbde01d6c00 con 0x7fbde013f210 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: _renew_subs 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:22:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.203:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fbde01d6fa0 con 0x7fbde013f210 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.5 v1:172.21.15.203:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdd8003c50 con 0x7fbde013f210 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.5 v1:172.21.15.203:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbdd80041c0 con 0x7fbde013f210 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbddd7fa640 4 set_mon_vals no callback set 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbdc7fff640 10 monclient: dump: 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.122 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.100+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbdc00065d0 con 0x7fbdc0002870 2024-05-21T17:22:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 600793750 0 0) 0x7fbdc00065d0 con 0x7fbdc0002870 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient(hunting): my global_id is 4407 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde00a0960 con 0x7fbdc0002870 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.5 v1:172.21.15.203:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fbdd80043e0 con 0x7fbde013f210 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3420228470 0 0) 0x7fbde00a0960 con 0x7fbdc0002870 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdc00065d0 con 0x7fbdc0002870 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.5 v1:172.21.15.203:6791/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7fbdd8046200 con 0x7fbde013f210 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbdd0001770 con 0x7fbdc0002870 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient: dump: 2024-05-21T17:22:01.124 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3491472071 0 0) 0x7fbdc00065d0 con 0x7fbdc0002870 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.3:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdc000a170 con 0x7fbdc0002870 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdde7fc640 10 monclient: get_auth_request con 0x7fbdcc038b70 auth_method 0 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbdd0001bf0 con 0x7fbdc0002870 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 >> v1:172.21.15.3:6792/0 conn(0x7fbdc0002870 legacy=0x7fbdc0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.104+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 >> v1:172.21.15.3:6792/0 conn(0x7fbdc0002870 legacy=0x7fbdc0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.108+0000 7fbde58c8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.108+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbdc0009e80 con 0x7fbdc0009140 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.108+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 76673217 0 0) 0x7fbdc0009e80 con 0x7fbdc0009140 2024-05-21T17:22:01.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.108+0000 7fbdc7fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.108+0000 7fbdc7fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.108+0000 7fbdc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.108+0000 7fbdc7fff640 10 monclient(hunting): my global_id is 4416 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.108+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc00065d0 con 0x7fbdc0009140 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1155787227 0 0) 0x7fbdc00065d0 con 0x7fbdc0009140 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdc0009e80 con 0x7fbdc0009140 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbdd0003c30 con 0x7fbdc0009140 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 10 monclient: dump: 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 599563693 0 0) 0x7fbdc0009e80 con 0x7fbdc0009140 2024-05-21T17:22:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 --> v1:172.21.15.3:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbdc000c420 con 0x7fbdc0009140 2024-05-21T17:22:01.129 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:22:01.129 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:22:01.129 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:22:01.129 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:22:01.129 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 16, 2024-05-21T17:22:01.129 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 45035, 2024-05-21T17:22:01.129 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:22:01.129 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 0, 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.130 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.131 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.131 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.131 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.131 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.131 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.131 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.131 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.131 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.132 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.132 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.132 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:22:01.134 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.135 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.136 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:22:01.137 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:22:01.137 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.137 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.137 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.137 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:22:01.138 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.138 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.138 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.138 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.138 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.138 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.138 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.139 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.142 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.143 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:22:01.144 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.145 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:22:01.146 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.148 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:22:01.148 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:22:01.148 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.148 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.148 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.148 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.149 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.150 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:22:01.150 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:22:01.150 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.150 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.150 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.151 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:22:01.151 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.151 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.151 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.151 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.151 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.151 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.152 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.153 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.153 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.153 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.153 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:22:01.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.153 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.153 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.154 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.154 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.154 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.154 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:22:01.154 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:22:01.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 <== mon.6 v1:172.21.15.3:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7512 (unknown 0 0 1290742599) 0x7fbdd00057d0 con 0x7fbdc0009140 2024-05-21T17:22:01.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:22:01.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:22:01.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 >> v1:172.21.15.3:6792/0 conn(0x7fbdc0009140 legacy=0x7fbdc00095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbdc7fff640 1 -- 172.21.15.3:0/1957356715 >> v1:172.21.15.3:6792/0 conn(0x7fbdc0009140 legacy=0x7fbdc00095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 >> v1:172.21.15.203:6812/343827626 conn(0x7fbdcc038b70 legacy=0x7fbdcc03b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbde58c8640 10 monclient: shutdown 2024-05-21T17:22:01.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbde58c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 >> v1:172.21.15.203:6791/0 conn(0x7fbde013f210 legacy=0x7fbde01d54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbddf7fe640 1 -- 172.21.15.3:0/1957356715 reap_dead start 2024-05-21T17:22:01.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbddf7fe640 1 -- 172.21.15.3:0/1957356715 reap_dead start 2024-05-21T17:22:01.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 shutdown_connections 2024-05-21T17:22:01.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 >> 172.21.15.3:0/1957356715 conn(0x7fbde00ab5b0 msgr2=0x7fbde0141600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 shutdown_connections 2024-05-21T17:22:01.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.112+0000 7fbde58c8640 1 -- 172.21.15.3:0/1957356715 wait complete. 2024-05-21T17:22:01.158 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-05-21T17:22:01.159 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-05-21T17:22:01.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-05-21T17:22:01.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient: get_monmap_and_config 2024-05-21T17:22:01.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient: monmap: 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:22:01.284784+0000 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:22:01.284784+0000 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.286 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 Processor -- start 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- start start 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient: init 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient: _add_conns ranks=[0,2,4,7,5,3,6,8,1] 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient(hunting): picked mon.noname-a con 0x7fb1ec135cf0 addr v1:172.21.15.3:6789/0 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient(hunting): picked mon.noname-b con 0x7fb1ec09cf90 addr v1:172.21.15.3:6790/0 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient(hunting): picked mon.noname-c con 0x7fb1ec09f9c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec09ecb0 con 0x7fb1ec135cf0 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec09e410 con 0x7fb1ec09cf90 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec09ccc0 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient(hunting): authenticate will time out at 988.442240s 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1ea7fc640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7fb1ec09cf90 legacy=0x7fb1ec0a35c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1eb7fe640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fb1ec09f9c0 0x7fb1ec0a3cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:38436/0 (socket says 172.21.15.3:38436) 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1eb7fe640 1 -- 172.21.15.3:0/3485387693 learned_addr learned my addr 172.21.15.3:0/3485387693 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1eaffd640 1 --1- 172.21.15.3:0/3485387693 >> v1:172.21.15.3:6789/0 conn(0x7fb1ec135cf0 0x7fb1ec0a2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:44004/0 (socket says 172.21.15.3:44004) 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888377487 0 0) 0x7fb1ec09ecb0 con 0x7fb1ec135cf0 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient(hunting): my global_id is 4545 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d4003530 con 0x7fb1ec135cf0 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1638873901 0 0) 0x7fb1ec09ccc0 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient(hunting): my global_id is 4540 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1ec09ecb0 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1041234032 0 0) 0x7fb1d4003530 con 0x7fb1ec135cf0 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 >> v1:172.21.15.3:6791/0 conn(0x7fb1ec09f9c0 legacy=0x7fb1ec0a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 >> v1:172.21.15.3:6790/0 conn(0x7fb1ec09cf90 legacy=0x7fb1ec0a35c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 monclient: found mon.noname-a 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:22:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec1360d0 con 0x7fb1ec135cf0 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb1e4001bc0 con 0x7fb1ec135cf0 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 5 monclient: authenticate success, global_id 4545 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: dump: 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1e4001d60 con 0x7fb1ec135cf0 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 1 -- 172.21.15.3:0/3485387693 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb1e4004eb0 con 0x7fb1ec135cf0 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1e9ffb640 10 monclient: dump: 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient: get_monmap_and_config success 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 4 set_mon_vals no callback set 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 10 monclient: shutdown 2024-05-21T17:22:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- 172.21.15.3:0/3485387693 >> v1:172.21.15.3:6789/0 conn(0x7fb1ec135cf0 legacy=0x7fb1ec0a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- 172.21.15.3:0/3485387693 shutdown_connections 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- 172.21.15.3:0/3485387693 >> 172.21.15.3:0/3485387693 conn(0x7fb1ec06c8d0 msgr2=0x7fb1ec06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- 172.21.15.3:0/3485387693 shutdown_connections 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.280+0000 7fb1f192f640 1 -- 172.21.15.3:0/3485387693 wait complete. 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: monmap: 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 1 Processor -- start 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 1 -- start start 2024-05-21T17:22:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: init 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: _add_conns ranks=[6,0,7,1,2,3,4,5,8] 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient(hunting): picked mon.noname-d con 0x7fb1ec09cf90 addr v1:172.21.15.3:6792/0 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient(hunting): picked mon.noname-a con 0x7fb1ec09f9c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient(hunting): picked mon.noname-i con 0x7fb1ec135cf0 addr v1:172.21.15.203:6792/0 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec070e70 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec071150 con 0x7fb1ec09cf90 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec13e820 con 0x7fb1ec135cf0 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient(hunting): authenticate will time out at 988.445291s 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1ea7fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb1ec09f9c0 0x7fb1ec13c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:44006/0 (socket says 172.21.15.3:44006) 2024-05-21T17:22:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1ea7fc640 1 -- 172.21.15.3:0/2116791751 learned_addr learned my addr 172.21.15.3:0/2116791751 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 508229477 0 0) 0x7fb1ec070e70 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): my global_id is 4554 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d8003530 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4205658082 0 0) 0x7fb1ec071150 con 0x7fb1ec09cf90 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): my global_id is 4425 2024-05-21T17:22:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1ec070e70 con 0x7fb1ec09cf90 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3694745384 0 0) 0x7fb1d8003530 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1ec071150 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb1e0004400 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): dump: 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:22:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2817195451 0 0) 0x7fb1ec070e70 con 0x7fb1ec09cf90 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1d8003530 con 0x7fb1ec09cf90 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb1e4004b70 con 0x7fb1ec09cf90 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): dump: 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616046948 0 0) 0x7fb1ec13e820 con 0x7fb1ec135cf0 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): my global_id is 4606 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1ec070e70 con 0x7fb1ec135cf0 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 325819028 0 0) 0x7fb1ec071150 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 >> v1:172.21.15.203:6792/0 conn(0x7fb1ec135cf0 legacy=0x7fb1ec1d5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 >> v1:172.21.15.3:6792/0 conn(0x7fb1ec09cf90 legacy=0x7fb1ec13c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 monclient: found mon.a 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec09d690 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 5 monclient: authenticate success, global_id 4554 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: _renew_subs 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec13dcc0 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: _renew_subs 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1ec13d0a0 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1e0004620 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb1e0004b90 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:22:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1e97fa640 4 set_mon_vals no callback set 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient: dump: 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb1cc0065d0 con 0x7fb1cc002870 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fb1e0004d30 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (unknown 2027697376 0 0) 0x7fb1e0045dc0 con 0x7fb1ec09f9c0 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762175381 0 0) 0x7fb1cc0065d0 con 0x7fb1cc002870 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 10 monclient(hunting): my global_id is 4535 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.284+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1ec071150 con 0x7fb1cc002870 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1eb7fe640 10 monclient: get_auth_request con 0x7fb1d80393c0 auth_method 0 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2714474480 0 0) 0x7fb1ec071150 con 0x7fb1cc002870 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1cc0065d0 con 0x7fb1cc002870 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb1e4005300 con 0x7fb1cc002870 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 10 monclient: dump: 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4271637622 0 0) 0x7fb1cc0065d0 con 0x7fb1cc002870 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1cc00a170 con 0x7fb1cc002870 2024-05-21T17:22:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb1e4007540 con 0x7fb1cc002870 2024-05-21T17:22:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:22:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 >> v1:172.21.15.3:6793/0 conn(0x7fb1cc002870 legacy=0x7fb1cc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.288+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 >> v1:172.21.15.3:6793/0 conn(0x7fb1cc002870 legacy=0x7fb1cc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1f192f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb1cc009e80 con 0x7fb1cc009140 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3097868052 0 0) 0x7fb1cc009e80 con 0x7fb1cc009140 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 10 monclient(hunting): my global_id is 4544 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1cc0065d0 con 0x7fb1cc009140 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2671710501 0 0) 0x7fb1cc0065d0 con 0x7fb1cc009140 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1cc009e80 con 0x7fb1cc009140 2024-05-21T17:22:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fb1e4006e40 con 0x7fb1cc009140 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.292+0000 7fb1d3fff640 10 monclient: dump: 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 657523087 0 0) 0x7fb1cc009e80 con 0x7fb1cc009140 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 --> v1:172.21.15.3:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb1cc00c420 con 0x7fb1cc009140 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 <== mon.8 v1:172.21.15.3:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7512 (unknown 0 0 3236659368) 0x7fb1e4006f30 con 0x7fb1cc009140 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1d3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1d3fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 >> v1:172.21.15.3:6793/0 conn(0x7fb1cc009140 legacy=0x7fb1cc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1d3fff640 1 -- 172.21.15.3:0/2116791751 >> v1:172.21.15.3:6793/0 conn(0x7fb1cc009140 legacy=0x7fb1cc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 16, 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 45227, 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:22:01.322 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.323 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:22:01.325 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:22:01.326 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.327 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:22:01.328 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:22:01.329 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.330 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.332 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.333 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.334 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:22:01.335 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.336 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.338 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.339 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.339 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.339 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.339 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.339 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:22:01.339 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.340 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:22:01.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.342 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.342 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.342 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.342 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.342 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.342 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:22:01.342 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:22:01.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 >> v1:172.21.15.203:6812/343827626 conn(0x7fb1d80393c0 legacy=0x7fb1d803b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1f192f640 10 monclient: shutdown 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1f192f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 >> v1:172.21.15.3:6789/0 conn(0x7fb1ec09f9c0 legacy=0x7fb1ec13c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1eb7fe640 1 -- 172.21.15.3:0/2116791751 reap_dead start 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1eb7fe640 1 -- 172.21.15.3:0/2116791751 reap_dead start 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 shutdown_connections 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 >> 172.21.15.3:0/2116791751 conn(0x7fb1ec06c8d0 msgr2=0x7fb1ec0a6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 shutdown_connections 2024-05-21T17:22:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.296+0000 7fb1f192f640 1 -- 172.21.15.3:0/2116791751 wait complete. 2024-05-21T17:22:01.344 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-05-21T17:22:01.345 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon g 2024-05-21T17:22:01.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-05-21T17:22:01.345 INFO:teuthology.orchestra.run.smithi203.stdout:process_2_[26983]: starting. 2024-05-21T17:22:01.345 INFO:teuthology.orchestra.run.smithi203.stdout:process_2_[26983]: deleting pool ceph_test_rados_delete_pools_parallel.smithi203-26981 2024-05-21T17:22:01.346 INFO:teuthology.orchestra.run.smithi203.stdout:process_2_[26983]: shutting down. 2024-05-21T17:22:01.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient: get_monmap_and_config 2024-05-21T17:22:01.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient: monmap: 2024-05-21T17:22:01.465 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.465 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.465 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:22:01.465323+0000 2024-05-21T17:22:01.465 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:22:01.465323+0000 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.466 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 1 Processor -- start 2024-05-21T17:22:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 1 -- start start 2024-05-21T17:22:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient: init 2024-05-21T17:22:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient: _add_conns ranks=[1,6,4,7,2,0,8,3,5] 2024-05-21T17:22:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient(hunting): picked mon.noname-f con 0x7fc340137a80 addr v1:172.21.15.203:6789/0 2024-05-21T17:22:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient(hunting): picked mon.noname-d con 0x7fc34009daf0 addr v1:172.21.15.3:6792/0 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient(hunting): picked mon.noname-c con 0x7fc34009df50 addr v1:172.21.15.3:6791/0 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc340138570 con 0x7fc340137a80 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc340068e50 con 0x7fc34009df50 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34009ce30 con 0x7fc34009daf0 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient(hunting): authenticate will time out at 988.622921s 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33ffff640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fc34009df50 0x7fc340141200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:38450/0 (socket says 172.21.15.3:38450) 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33ffff640 1 -- 172.21.15.3:0/1539019937 learned_addr learned my addr 172.21.15.3:0/1539019937 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33f7fe640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fc340137a80 legacy=0x7fc340137e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3165722939 0 0) 0x7fc340068e50 con 0x7fc34009df50 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient(hunting): my global_id is 4567 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc32c003530 con 0x7fc34009df50 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 473024793 0 0) 0x7fc34009ce30 con 0x7fc34009daf0 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient(hunting): my global_id is 4443 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc340068e50 con 0x7fc34009daf0 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 857045780 0 0) 0x7fc32c003530 con 0x7fc34009df50 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 >> v1:172.21.15.3:6792/0 conn(0x7fc34009daf0 legacy=0x7fc34013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 >> v1:172.21.15.203:6789/0 conn(0x7fc340137a80 legacy=0x7fc340137e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 monclient: found mon.noname-c 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34009d110 con 0x7fc34009df50 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 5 monclient: authenticate success, global_id 4567 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc338001840 con 0x7fc34009df50 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: dump: 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.476 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3380019e0 con 0x7fc34009df50 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 10 monclient: get_monmap_and_config success 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 1 -- 172.21.15.3:0/1539019937 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc338004bf0 con 0x7fc34009df50 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc345fb0640 4 set_mon_vals no callback set 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.460+0000 7fc33e7fc640 10 monclient: dump: 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.478 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: shutdown 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- 172.21.15.3:0/1539019937 >> v1:172.21.15.3:6791/0 conn(0x7fc34009df50 legacy=0x7fc340141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- 172.21.15.3:0/1539019937 shutdown_connections 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- 172.21.15.3:0/1539019937 >> 172.21.15.3:0/1539019937 conn(0x7fc34006c8d0 msgr2=0x7fc34006cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- 172.21.15.3:0/1539019937 shutdown_connections 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- 172.21.15.3:0/1539019937 wait complete. 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: monmap: 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:22:01.480 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 Processor -- start 2024-05-21T17:22:01.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- start start 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: init 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: _add_conns ranks=[5,1,0,8,7,2,6,3,4] 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient(hunting): picked mon.noname-h con 0x7fc34009daf0 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient(hunting): picked mon.noname-f con 0x7fc34009df50 addr v1:172.21.15.203:6789/0 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient(hunting): picked mon.noname-a con 0x7fc340137a80 addr v1:172.21.15.3:6789/0 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3401356d0 con 0x7fc340137a80 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc340135fb0 con 0x7fc34009df50 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc340136450 con 0x7fc34009daf0 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient(hunting): authenticate will time out at 988.626031s 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc33ffff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc340137a80 0x7fc3401d5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:44022/0 (socket says 172.21.15.3:44022) 2024-05-21T17:22:01.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc33ffff640 1 -- 172.21.15.3:0/1204774502 learned_addr learned my addr 172.21.15.3:0/1204774502 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc33effd640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.203:6789/0 conn(0x7fc34009df50 legacy=0x7fc340141d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 338796705 0 0) 0x7fc3401356d0 con 0x7fc340137a80 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): my global_id is 4617 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc320003530 con 0x7fc340137a80 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3438218187 0 0) 0x7fc320003530 con 0x7fc340137a80 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3401356d0 con 0x7fc340137a80 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc338001800 con 0x7fc340137a80 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): dump: 2024-05-21T17:22:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2269049571 0 0) 0x7fc340136450 con 0x7fc34009daf0 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient(hunting): my global_id is 4613 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc320003530 con 0x7fc34009daf0 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1926689543 0 0) 0x7fc3401356d0 con 0x7fc340137a80 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.203:6791/0 conn(0x7fc34009daf0 legacy=0x7fc3401cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.203:6789/0 conn(0x7fc34009df50 legacy=0x7fc340141d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 monclient: found mon.a 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc340142470 con 0x7fc340137a80 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 5 monclient: authenticate success, global_id 4617 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: _renew_subs 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc340142720 con 0x7fc340137a80 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: _renew_subs 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3401d6b00 con 0x7fc340137a80 2024-05-21T17:22:01.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3380019a0 con 0x7fc340137a80 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc338004c00 con 0x7fc340137a80 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc33dffb640 4 set_mon_vals no callback set 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 10 monclient: dump: 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.487 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3240065d0 con 0x7fc324002870 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.464+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fc338010210 con 0x7fc340137a80 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3506+0+0 (unknown 1282225426 0 0) 0x7fc338045ba0 con 0x7fc340137a80 2024-05-21T17:22:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc33effd640 10 monclient: get_auth_request con 0x7fc320038c50 auth_method 0 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1563936495 0 0) 0x7fc3240065d0 con 0x7fc324002870 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient(hunting): my global_id is 4584 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3401356d0 con 0x7fc324002870 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2396494318 0 0) 0x7fc3401356d0 con 0x7fc324002870 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3240065d0 con 0x7fc324002870 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc3300042d0 con 0x7fc324002870 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient: dump: 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.489 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.490 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1262667127 0 0) 0x7fc3240065d0 con 0x7fc324002870 2024-05-21T17:22:01.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc32400a170 con 0x7fc324002870 2024-05-21T17:22:01.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc3300045a0 con 0x7fc324002870 2024-05-21T17:22:01.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:22:01.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.203:6790/0 conn(0x7fc324002870 legacy=0x7fc324004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.468+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.203:6790/0 conn(0x7fc324002870 legacy=0x7fc324004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:22:01.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc324009e80 con 0x7fc324009140 2024-05-21T17:22:01.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556858168 0 0) 0x7fc324009e80 con 0x7fc324009140 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient(hunting): my global_id is 4593 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3240065d0 con 0x7fc324009140 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3252765045 0 0) 0x7fc3240065d0 con 0x7fc324009140 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc324009e80 con 0x7fc324009140 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc330004d40 con 0x7fc324009140 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient: dump: 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.493 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3933980008 0 0) 0x7fc324009e80 con 0x7fc324009140 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 --> v1:172.21.15.203:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc32400c420 con 0x7fc324009140 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 <== mon.3 v1:172.21.15.203:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7512 (unknown 0 0 3024076840) 0x7fc3300061b0 con 0x7fc324009140 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.203:6790/0 conn(0x7fc324009140 legacy=0x7fc3240095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc337fff640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.203:6790/0 conn(0x7fc324009140 legacy=0x7fc3240095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:22:01.495 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 16, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 45383, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:22:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:22:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:22:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:22:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:22:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.515 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.515 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.515 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:22:01.515 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:22:01.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.203:6812/343827626 conn(0x7fc320038c50 legacy=0x7fc32003b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 10 monclient: shutdown 2024-05-21T17:22:01.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 >> v1:172.21.15.3:6789/0 conn(0x7fc340137a80 legacy=0x7fc3401d5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc33ffff640 1 -- 172.21.15.3:0/1204774502 reap_dead start 2024-05-21T17:22:01.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc33ffff640 1 -- 172.21.15.3:0/1204774502 reap_dead start 2024-05-21T17:22:01.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 shutdown_connections 2024-05-21T17:22:01.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 >> 172.21.15.3:0/1204774502 conn(0x7fc34006c8d0 msgr2=0x7fc340137e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 shutdown_connections 2024-05-21T17:22:01.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.476+0000 7fc345fb0640 1 -- 172.21.15.3:0/1204774502 wait complete. 2024-05-21T17:22:01.517 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-05-21T17:22:01.517 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-05-21T17:22:01.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-05-21T17:22:01.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient: get_monmap_and_config 2024-05-21T17:22:01.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient: monmap: 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:22:01.640825+0000 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:22:01.640825+0000 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.642 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.642 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.642 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.642 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 1 Processor -- start 2024-05-21T17:22:01.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 1 -- start start 2024-05-21T17:22:01.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient: init 2024-05-21T17:22:01.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient: _add_conns ranks=[7,2,5,6,4,1,3,0,8] 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient(hunting): picked mon.noname-i con 0x7f68b81388b0 addr v1:172.21.15.203:6792/0 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient(hunting): picked mon.noname-b con 0x7f68b81393a0 addr v1:172.21.15.3:6790/0 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient(hunting): picked mon.noname-h con 0x7f68b809db30 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68b809e720 con 0x7f68b81393a0 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68b80a41b0 con 0x7f68b809db30 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68b80a43f0 con 0x7f68b81388b0 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 10 monclient(hunting): authenticate will time out at 988.798267s 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b67fc640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f68b81393a0 legacy=0x7f68b813de50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b77fe640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f68b809db30 0x7f68b8141580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.3:38248/0 (socket says 172.21.15.3:38248) 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b77fe640 1 -- 172.21.15.3:0/2362414919 learned_addr learned my addr 172.21.15.3:0/2362414919 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855056579 0 0) 0x7f68b80a41b0 con 0x7f68b809db30 2024-05-21T17:22:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient(hunting): my global_id is 4622 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68a0003530 con 0x7f68b809db30 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2126147076 0 0) 0x7f68b80a43f0 con 0x7f68b81388b0 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient(hunting): my global_id is 4624 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68b80a41b0 con 0x7f68b81388b0 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2942450912 0 0) 0x7f68a0003530 con 0x7f68b809db30 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 >> v1:172.21.15.203:6792/0 conn(0x7f68b81388b0 legacy=0x7f68b8138c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 >> v1:172.21.15.3:6790/0 conn(0x7f68b81393a0 legacy=0x7f68b813de50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 monclient: found mon.noname-h 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.203:6791/0 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 --> v1:172.21.15.203:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b8069140 con 0x7f68b809db30 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f68b0001bc0 con 0x7f68b809db30 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68b5ffb640 10 monclient: dump: 2024-05-21T17:22:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 5 monclient: authenticate success, global_id 4622 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.636+0000 7f68bd424640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 <== mon.5 v1:172.21.15.203:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f68b0001d60 con 0x7f68b809db30 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b5ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b5ffb640 1 -- 172.21.15.3:0/2362414919 <== mon.5 v1:172.21.15.203:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f68b0004f70 con 0x7f68b809db30 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b5ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b5ffb640 10 monclient: dump: 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: get_monmap_and_config success 2024-05-21T17:22:01.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 4 set_mon_vals no callback set 2024-05-21T17:22:01.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: shutdown 2024-05-21T17:22:01.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- 172.21.15.3:0/2362414919 >> v1:172.21.15.203:6791/0 conn(0x7f68b809db30 legacy=0x7f68b8141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- 172.21.15.3:0/2362414919 shutdown_connections 2024-05-21T17:22:01.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- 172.21.15.3:0/2362414919 >> 172.21.15.3:0/2362414919 conn(0x7f68b806c8d0 msgr2=0x7f68b806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- 172.21.15.3:0/2362414919 shutdown_connections 2024-05-21T17:22:01.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- 172.21.15.3:0/2362414919 wait complete. 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: monmap: 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:22:01.656 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 Processor -- start 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- start start 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: init 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: _add_conns ranks=[2,5,1,4,7,0,8,6,3] 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient(hunting): picked mon.noname-b con 0x7f68b809db30 addr v1:172.21.15.3:6790/0 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient(hunting): picked mon.noname-h con 0x7f68b81388b0 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient(hunting): picked mon.noname-f con 0x7f68b81393a0 addr v1:172.21.15.203:6789/0 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68b813b4b0 con 0x7f68b81393a0 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68b813c230 con 0x7f68b809db30 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68b813cf00 con 0x7f68b81388b0 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient(hunting): authenticate will time out at 988.802150s 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b6ffd640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f68b809db30 legacy=0x7f68b8141ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.3:6790/0 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b77fe640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f68b81393a0 legacy=0x7f68b81d5b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b67fc640 1 --1- >> v1:172.21.15.203:6791/0 conn(0x7f68b81388b0 0x7f68b81d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6791/0 says I am v1:172.21.15.3:38254/0 (socket says 172.21.15.3:38254) 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68b67fc640 1 -- 172.21.15.3:0/2764998719 learned_addr learned my addr 172.21.15.3:0/2764998719 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3921153559 0 0) 0x7f68b813cf00 con 0x7f68b81388b0 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient(hunting): my global_id is 4631 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68a4003530 con 0x7f68b81388b0 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3788482504 0 0) 0x7f68a4003530 con 0x7f68b81388b0 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68b813cf00 con 0x7f68b81388b0 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f68ac004400 con 0x7f68b81388b0 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient(hunting): dump: 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.659 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.660 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 951477309 0 0) 0x7f68b813cf00 con 0x7f68b81388b0 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 >> v1:172.21.15.3:6790/0 conn(0x7f68b809db30 legacy=0x7f68b8141ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 >> v1:172.21.15.203:6789/0 conn(0x7f68b81393a0 legacy=0x7f68b81d5b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 monclient: found mon.h 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b81423f0 con 0x7f68b81388b0 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f689f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 5 monclient: authenticate success, global_id 4631 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: _renew_subs 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f68b8142620 con 0x7f68b81388b0 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: _renew_subs 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.203:6791/0 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.640+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f68b81cd500 con 0x7f68b81388b0 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f68ac004620 con 0x7f68b81388b0 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f68ac004c50 con 0x7f68b81388b0 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f68b4ff9640 4 set_mon_vals no callback set 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: dump: 2024-05-21T17:22:01.662 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.663 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f68bd424640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f68980065d0 con 0x7f6898002870 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f68ac004e70 con 0x7f68b81388b0 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 8 ==== osd_map(18..18 src has 1..18) ==== 3506+0+0 (unknown 1282225426 0 0) 0x7f68ac045ca0 con 0x7f68b81388b0 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405878181 0 0) 0x7f68980065d0 con 0x7f6898002870 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient(hunting): my global_id is 4640 2024-05-21T17:22:01.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68b813cf00 con 0x7f6898002870 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f68b77fe640 10 monclient: get_auth_request con 0x7f68a4038a70 auth_method 0 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 205859781 0 0) 0x7f68b813cf00 con 0x7f6898002870 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68980065d0 con 0x7f6898002870 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f68a80018a0 con 0x7f6898002870 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: dump: 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.665 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3128861322 0 0) 0x7f68980065d0 con 0x7f6898002870 2024-05-21T17:22:01.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f689800a170 con 0x7f6898002870 2024-05-21T17:22:01.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f68a8001c00 con 0x7f6898002870 2024-05-21T17:22:01.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:22:01.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 >> v1:172.21.15.203:6791/0 conn(0x7f6898002870 legacy=0x7f6898004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.644+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 >> v1:172.21.15.203:6791/0 conn(0x7f6898002870 legacy=0x7f6898004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f68bd424640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:22:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6898009e80 con 0x7f6898009140 2024-05-21T17:22:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171341323 0 0) 0x7f6898009e80 con 0x7f6898009140 2024-05-21T17:22:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 10 monclient(hunting): my global_id is 4649 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68980065d0 con 0x7f6898009140 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3832319251 0 0) 0x7f68980065d0 con 0x7f6898009140 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6898009e80 con 0x7f6898009140 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f68a8003c70 con 0x7f6898009140 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 10 monclient: dump: 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.670 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1593000369 0 0) 0x7f6898009e80 con 0x7f6898009140 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.652+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 --> v1:172.21.15.203:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f689800c420 con 0x7f6898009140 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 <== mon.5 v1:172.21.15.203:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7682 (unknown 0 0 4021640929) 0x7f68a80057d0 con 0x7f6898009140 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f689f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f689f7fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 >> v1:172.21.15.203:6791/0 conn(0x7f6898009140 legacy=0x7f68980095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f689f7fe640 1 -- 172.21.15.3:0/2764998719 >> v1:172.21.15.203:6791/0 conn(0x7f6898009140 legacy=0x7f68980095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:22:01.671 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 16, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 45559, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:22:01.672 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.673 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.674 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:22:01.675 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.676 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.677 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:22:01.679 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.680 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.682 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:22:01.683 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.684 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.685 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.686 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.688 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.689 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.692 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.692 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 4 2024-05-21T17:22:01.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.692 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.692 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:22:01.692 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:22:01.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 >> v1:172.21.15.203:6812/343827626 conn(0x7f68a4038a70 legacy=0x7f68a403af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68bd424640 10 monclient: shutdown 2024-05-21T17:22:01.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68bd424640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 >> v1:172.21.15.203:6791/0 conn(0x7f68b81388b0 legacy=0x7f68b81d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68b77fe640 1 -- 172.21.15.3:0/2764998719 reap_dead start 2024-05-21T17:22:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68b77fe640 1 -- 172.21.15.3:0/2764998719 reap_dead start 2024-05-21T17:22:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 shutdown_connections 2024-05-21T17:22:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 >> 172.21.15.3:0/2764998719 conn(0x7f68b806c8d0 msgr2=0x7f68b80a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 shutdown_connections 2024-05-21T17:22:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.656+0000 7f68bd424640 1 -- 172.21.15.3:0/2764998719 wait complete. 2024-05-21T17:22:01.694 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-05-21T17:22:01.694 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-05-21T17:22:01.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-05-21T17:22:01.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient: get_monmap_and_config 2024-05-21T17:22:01.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient: monmap: 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:22:01.812529+0000 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:22:01.812529+0000 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.814 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.814 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.814 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.814 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.814 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.814 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.814 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 1 Processor -- start 2024-05-21T17:22:01.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 1 -- start start 2024-05-21T17:22:01.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient: init 2024-05-21T17:22:01.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient: _add_conns ranks=[5,4,6,7,0,8,1,2,3] 2024-05-21T17:22:01.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient(hunting): picked mon.noname-h con 0x7f72280a4730 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:01.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient(hunting): picked mon.noname-c con 0x7f72280a5d40 addr v1:172.21.15.3:6791/0 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient(hunting): picked mon.noname-d con 0x7f722813f2f0 addr v1:172.21.15.3:6792/0 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72280a6f50 con 0x7f72280a5d40 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72280a2960 con 0x7f72280a4730 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72280a2c40 con 0x7f722813f2f0 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient(hunting): authenticate will time out at 988.970052s 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722dbc3640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f72280a5d40 0x7f722813dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:38460/0 (socket says 172.21.15.3:38460) 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722dbc3640 1 -- 172.21.15.3:0/1637244558 learned_addr learned my addr 172.21.15.3:0/1637244558 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3436750292 0 0) 0x7f72280a6f50 con 0x7f72280a5d40 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient(hunting): my global_id is 4576 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7214003530 con 0x7f72280a5d40 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2233621815 0 0) 0x7f72280a2c40 con 0x7f722813f2f0 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient(hunting): my global_id is 4452 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72280a6f50 con 0x7f722813f2f0 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4010374979 0 0) 0x7f7214003530 con 0x7f72280a5d40 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 >> v1:172.21.15.3:6792/0 conn(0x7f722813f2f0 legacy=0x7f72281416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 >> v1:172.21.15.203:6791/0 conn(0x7f72280a4730 legacy=0x7f72280a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 monclient: found mon.noname-c 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72280a5220 con 0x7f72280a5d40 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7218001840 con 0x7f72280a5d40 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 5 monclient: authenticate success, global_id 4576 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: dump: 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.826 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7218001a60 con 0x7f72280a5d40 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient: get_monmap_and_config success 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 4 set_mon_vals no callback set 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 1 -- 172.21.15.3:0/1637244558 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7218001d50 con 0x7f72280a5d40 2024-05-21T17:22:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f722d3c2640 10 monclient: dump: 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 10 monclient: shutdown 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 1 -- 172.21.15.3:0/1637244558 >> v1:172.21.15.3:6791/0 conn(0x7f72280a5d40 legacy=0x7f722813dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 1 -- 172.21.15.3:0/1637244558 shutdown_connections 2024-05-21T17:22:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.808+0000 7f7230628640 1 -- 172.21.15.3:0/1637244558 >> 172.21.15.3:0/1637244558 conn(0x7f72280ab610 msgr2=0x7f72280a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- 172.21.15.3:0/1637244558 shutdown_connections 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- 172.21.15.3:0/1637244558 wait complete. 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: build_initial_monmap 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: monmap: 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:22:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 Processor -- start 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- start start 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: init 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: _reopen_session rank -1 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: _add_conns ranks=[8,1,5,7,6,2,0,3,4] 2024-05-21T17:22:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient(hunting): picked mon.noname-e con 0x7f72280a4730 addr v1:172.21.15.3:6793/0 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient(hunting): picked mon.noname-f con 0x7f72280a5d40 addr v1:172.21.15.203:6789/0 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient(hunting): picked mon.noname-h con 0x7f722813f2f0 addr v1:172.21.15.203:6791/0 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72280a1d40 con 0x7f72280a5d40 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72281ce1e0 con 0x7f722813f2f0 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72281335b0 con 0x7f72280a4730 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient(hunting): _renew_subs 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient(hunting): authenticate will time out at 988.973748s 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f722dbc3640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f72280a5d40 legacy=0x7f72281cdad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f722e3c4640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f72280a4730 0x7f72281cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:38464/0 (socket says 172.21.15.3:38464) 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f722e3c4640 1 -- 172.21.15.3:0/2623669563 learned_addr learned my addr 172.21.15.3:0/2623669563 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 5724736 0 0) 0x7f72281335b0 con 0x7f72280a4730 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): my global_id is 4553 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f720c003530 con 0x7f72280a4730 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3371430289 0 0) 0x7f720c003530 con 0x7f72280a4730 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72281335b0 con 0x7f72280a4730 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2934368084 0 0) 0x7f72281ce1e0 con 0x7f722813f2f0 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): my global_id is 4667 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f720c003530 con 0x7f722813f2f0 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7224001b40 con 0x7f72280a4730 2024-05-21T17:22:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): dump: 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4040269730 0 0) 0x7f72281335b0 con 0x7f72280a4730 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 >> v1:172.21.15.203:6791/0 conn(0x7f722813f2f0 legacy=0x7f72281d5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 >> v1:172.21.15.203:6789/0 conn(0x7f72280a5d40 legacy=0x7f72281cdad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient: _finish_hunting 0 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 monclient: found mon.e 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:22:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72281355a0 con 0x7f72280a4730 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient: _finish_auth 0 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient: _check_auth_tickets 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 5 monclient: authenticate success, global_id 4553 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: _renew_subs 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7228133e10 con 0x7f72280a4730 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: _renew_subs 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f72281342c0 con 0x7f72280a4730 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7224001ce0 con 0x7f72280a4730 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:22:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7224003cd0 con 0x7f72280a4730 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f722cbc1640 4 set_mon_vals no callback set 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 10 monclient: dump: 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f72100065d0 con 0x7f7210002870 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.812+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f7224003ef0 con 0x7f72280a4730 2024-05-21T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(18..18 src has 1..18) ==== 3506+0+0 (unknown 1282225426 0 0) 0x7f7224045be0 con 0x7f72280a4730 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f722ebc5640 10 monclient: get_auth_request con 0x7f720c038cd0 auth_method 0 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609031089 0 0) 0x7f72100065d0 con 0x7f7210002870 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient(hunting): my global_id is 4633 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72281335b0 con 0x7f7210002870 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4020642249 0 0) 0x7f72281335b0 con 0x7f7210002870 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72100065d0 con 0x7f7210002870 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7218001800 con 0x7f7210002870 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient: dump: 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 12225950 0 0) 0x7f72100065d0 con 0x7f7210002870 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f721000a170 con 0x7f7210002870 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7218001c50 con 0x7f7210002870 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 >> v1:172.21.15.203:6792/0 conn(0x7f7210002870 legacy=0x7f7210004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.816+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 >> v1:172.21.15.203:6792/0 conn(0x7f7210002870 legacy=0x7f7210004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7210009e60 con 0x7f7210009120 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1059576698 0 0) 0x7f7210009e60 con 0x7f7210009120 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient(hunting): _negotiate 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient(hunting): my global_id is 4642 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72100065d0 con 0x7f7210009120 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2369658051 0 0) 0x7f72100065d0 con 0x7f7210009120 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7210009e60 con 0x7f7210009120 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7218001080 con 0x7f7210009120 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:22:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient: dump: 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:22:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3420290282 0 0) 0x7f7210009e60 con 0x7f7210009120 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 --> v1:172.21.15.203:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f721000c400 con 0x7f7210009120 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 <== mon.7 v1:172.21.15.203:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7852 (unknown 0 0 1623663210) 0x7f7218005340 con 0x7f7210009120 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 >> v1:172.21.15.203:6792/0 conn(0x7f7210009120 legacy=0x7f72100095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f721effd640 1 -- 172.21.15.3:0/2623669563 >> v1:172.21.15.203:6792/0 conn(0x7f7210009120 legacy=0x7f72100095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 16, 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 45707, 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:22:01.846 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.847 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.848 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:22:01.849 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:22:01.850 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:22:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.852 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:22:01.855 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.862 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.863 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:22:01.864 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.865 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.866 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.869 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": [ 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:22:01.870 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:22:01.871 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:22:01.871 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:22:01.871 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:22:01.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:22:01.871 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:22:01.871 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:22:01.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 >> v1:172.21.15.203:6812/343827626 conn(0x7f720c038cd0 legacy=0x7f720c03b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 10 monclient: shutdown 2024-05-21T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 >> v1:172.21.15.3:6793/0 conn(0x7f72280a4730 legacy=0x7f72281cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f722ebc5640 1 -- 172.21.15.3:0/2623669563 reap_dead start 2024-05-21T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f722ebc5640 1 -- 172.21.15.3:0/2623669563 reap_dead start 2024-05-21T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 shutdown_connections 2024-05-21T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 >> 172.21.15.3:0/2623669563 conn(0x7f72280ab610 msgr2=0x7f72280a35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 shutdown_connections 2024-05-21T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:22:01.824+0000 7f7230628640 1 -- 172.21.15.3:0/2623669563 wait complete. 2024-05-21T17:22:01.874 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-05-21T17:22:01.874 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-05-21T17:22:03.541 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-05-21T17:22:03.542 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[27016]: starting. 2024-05-21T17:22:03.542 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[27016]: creating pool ceph_test_rados_delete_pools_parallel.smithi203-26981 2024-05-21T17:22:03.542 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[27016]: created object 0... 2024-05-21T17:22:03.542 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[27016]: created object 25... 2024-05-21T17:22:03.542 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[27016]: created object 49... 2024-05-21T17:22:03.542 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[27016]: finishing. 2024-05-21T17:22:03.542 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[27016]: shutting down. 2024-05-21T17:22:03.570 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-05-21T17:22:03.570 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[27018]: starting. 2024-05-21T17:22:03.570 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[27018]: listing objects. 2024-05-21T17:22:03.570 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[27018]: listed object 0... 2024-05-21T17:22:03.571 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[27018]: listed object 25... 2024-05-21T17:22:03.571 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[27018]: saw 50 objects 2024-05-21T17:22:03.571 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[27018]: shutting down. 2024-05-21T17:22:04.381 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-05-21T17:22:04.381 INFO:teuthology.orchestra.run.smithi203.stdout:process_4_[27017]: starting. 2024-05-21T17:22:04.381 INFO:teuthology.orchestra.run.smithi203.stdout:process_4_[27017]: deleting pool ceph_test_rados_delete_pools_parallel.smithi203-26981 2024-05-21T17:22:04.381 INFO:teuthology.orchestra.run.smithi203.stdout:process_4_[27017]: shutting down. 2024-05-21T17:22:04.384 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-05-21T17:22:04.384 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-05-21T17:22:04.384 INFO:teuthology.orchestra.run.smithi203.stdout:******* SUCCESS ********** 2024-05-21T17:22:04.387 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-05-21T17:22:04.396 INFO:tasks.mon_thrash:joining mon_thrasher 2024-05-21T17:23:31.875 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-21T17:23:31.875 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-05-21T17:23:31.875 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-21T17:23:31.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-21T17:23:31.879 INFO:tasks.ceph.mon.b:Started 2024-05-21T17:23:31.879 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-05-21T17:23:31.879 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-05-21T17:23:31.879 INFO:tasks.ceph.mon.f:Restarting daemon 2024-05-21T17:23:31.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-05-21T17:23:31.882 INFO:tasks.ceph.mon.f:Started 2024-05-21T17:23:31.882 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-05-21T17:23:31.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-21T17:23:32.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient: get_monmap_and_config 2024-05-21T17:23:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient: build_initial_monmap 2024-05-21T17:23:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient: monmap: 2024-05-21T17:23:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:32.010381+0000 2024-05-21T17:23:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:32.010381+0000 2024-05-21T17:23:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:32.011 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 1 Processor -- start 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 1 -- start start 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient: init 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient: _add_conns ranks=[5,1,0,6,7,8,4,3,2] 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient(hunting): picked mon.noname-h con 0x7efea8135bf0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient(hunting): picked mon.noname-f con 0x7efea8135fd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient(hunting): picked mon.noname-a con 0x7efea813fc90 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea80a3b80 con 0x7efea813fc90 2024-05-21T17:23:32.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea8068f90 con 0x7efea8135fd0 2024-05-21T17:23:32.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea80a4650 con 0x7efea8135bf0 2024-05-21T17:23:32.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:32.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efead1e8640 10 monclient(hunting): authenticate will time out at 1079.167494s 2024-05-21T17:23:32.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efea759e640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efea813fc90 0x7efea8142080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:37456/0 (socket says 172.21.15.3:37456) 2024-05-21T17:23:32.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:32.004+0000 7efea759e640 1 -- 172.21.15.3:0/3481914906 learned_addr learned my addr 172.21.15.3:0/3481914906 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:35.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 10 monclient(hunting): tick 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 10 monclient(hunting): _check_auth_tickets 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 1 monclient(hunting): continuing hunt 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 10 monclient(hunting): _reopen_session rank -1 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 1 -- 172.21.15.3:0/3481914906 >> v1:172.21.15.203:6791/0 conn(0x7efea8135bf0 legacy=0x7efea80a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 1 -- 172.21.15.3:0/3481914906 >> v1:172.21.15.203:6789/0 conn(0x7efea8135fd0 legacy=0x7efea80a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 1 -- 172.21.15.3:0/3481914906 >> v1:172.21.15.3:6789/0 conn(0x7efea813fc90 legacy=0x7efea8142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 10 monclient: _add_conns ranks=[7,6,3,2,4,8] 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 10 monclient(hunting): picked mon.noname-i con 0x7efe940037f0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 10 monclient(hunting): picked mon.noname-d con 0x7efe94007480 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:35.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 10 monclient(hunting): picked mon.noname-g con 0x7efe9400b1b0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:35.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 1 -- 172.21.15.3:0/3481914906 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea809ce70 con 0x7efe9400b1b0 2024-05-21T17:23:35.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 1 -- 172.21.15.3:0/3481914906 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe9400fff0 con 0x7efe94007480 2024-05-21T17:23:35.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.004+0000 7efea4d99640 1 -- 172.21.15.3:0/3481914906 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe94011290 con 0x7efe940037f0 2024-05-21T17:23:35.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:35.008+0000 7efea4d99640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:36.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1506200894 0 0) 0x7efe94011290 con 0x7efe940037f0 2024-05-21T17:23:36.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:23:36.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient(hunting): my global_id is 4651 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe90003530 con 0x7efe940037f0 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204095395 0 0) 0x7efe9400fff0 con 0x7efe94007480 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient(hunting): my global_id is 4461 2024-05-21T17:23:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe94011290 con 0x7efe94007480 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1545823841 0 0) 0x7efe90003530 con 0x7efe940037f0 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 >> v1:172.21.15.3:6792/0 conn(0x7efe94007480 legacy=0x7efe940098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 >> v1:172.21.15.203:6790/0 conn(0x7efe9400b1b0 legacy=0x7efe9400d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: _finish_hunting 0 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 monclient: found mon.noname-i 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.203:6792/0 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 --> v1:172.21.15.203:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe940125f0 con 0x7efe940037f0 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea759e640 1 -- 172.21.15.3:0/3481914906 reap_dead start 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: _finish_auth 0 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: _check_auth_tickets 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:36.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efe9c0017b0 con 0x7efe940037f0 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: dump: 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:36.997 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efead1e8640 5 monclient: authenticate success, global_id 4651 2024-05-21T17:23:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efead1e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 <== mon.7 v1:172.21.15.203:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe9c0019c0 con 0x7efe940037f0 2024-05-21T17:23:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 1 -- 172.21.15.3:0/3481914906 <== mon.7 v1:172.21.15.203:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efe9c003960 con 0x7efe940037f0 2024-05-21T17:23:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efea5d9b640 10 monclient: dump: 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.000 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efead1e8640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.988+0000 7efead1e8640 4 set_mon_vals no callback set 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 10 monclient: shutdown 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 1 -- 172.21.15.3:0/3481914906 >> v1:172.21.15.203:6792/0 conn(0x7efe940037f0 legacy=0x7efe94005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 1 -- 172.21.15.3:0/3481914906 shutdown_connections 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 1 -- 172.21.15.3:0/3481914906 >> 172.21.15.3:0/3481914906 conn(0x7efea806c8d0 msgr2=0x7efea806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 1 -- 172.21.15.3:0/3481914906 shutdown_connections 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 1 -- 172.21.15.3:0/3481914906 wait complete. 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 10 monclient: monmap: 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 1 Processor -- start 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 1 -- start start 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.992+0000 7efead1e8640 10 monclient: init 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 10 monclient: _add_conns ranks=[8,0,3,6,4,5,7,1,2] 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 10 monclient(hunting): picked mon.noname-e con 0x7efe940037f0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 10 monclient(hunting): picked mon.noname-a con 0x7efea80a1220 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 10 monclient(hunting): picked mon.noname-g con 0x7efea80a1600 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea8070160 con 0x7efea80a1220 2024-05-21T17:23:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea80a21f0 con 0x7efea80a1600 2024-05-21T17:23:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea8071680 con 0x7efe940037f0 2024-05-21T17:23:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efead1e8640 10 monclient(hunting): authenticate will time out at 1084.156907s 2024-05-21T17:23:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efea659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efea80a1220 0x7efea81d72a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55032/0 (socket says 172.21.15.3:55032) 2024-05-21T17:23:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efea659c640 1 -- 172.21.15.3:0/2966528650 learned_addr learned my addr 172.21.15.3:0/2966528650 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367801165 0 0) 0x7efea8070160 con 0x7efea80a1220 2024-05-21T17:23:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efe8f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efe8f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efe8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efe8f7fe640 10 monclient(hunting): my global_id is 4626 2024-05-21T17:23:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:36.996+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe94005420 con 0x7efea80a1220 2024-05-21T17:23:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503228597 0 0) 0x7efea8071680 con 0x7efe940037f0 2024-05-21T17:23:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient(hunting): my global_id is 4562 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efea8070160 con 0x7efe940037f0 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 638723308 0 0) 0x7efea8070160 con 0x7efe940037f0 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efea8071680 con 0x7efe940037f0 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efe9c001810 con 0x7efe940037f0 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient(hunting): dump: 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2097498806 0 0) 0x7efea8071680 con 0x7efe940037f0 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 >> v1:172.21.15.203:6790/0 conn(0x7efea80a1600 legacy=0x7efea81da9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 >> v1:172.21.15.3:6789/0 conn(0x7efea80a1220 legacy=0x7efea81d72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 monclient: found mon.e 2024-05-21T17:23:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:37.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea8072340 con 0x7efe940037f0 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.056+0000 7efe8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 5 monclient: authenticate success, global_id 4562 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 10 monclient: _renew_subs 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efea813efc0 con 0x7efe940037f0 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 10 monclient: _renew_subs 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7efea80a1f60 con 0x7efe940037f0 2024-05-21T17:23:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe9c0019c0 con 0x7efe940037f0 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efe8f7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7efe9c003f50 con 0x7efe940037f0 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efe8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efe8f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efea559a640 4 set_mon_vals no callback set 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efe8f7fe640 10 monclient: dump: 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.074 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe88005090 con 0x7efe940037f0 2024-05-21T17:23:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7efe9c00f0d0 con 0x7efe940037f0 2024-05-21T17:23:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7efe9c044890 con 0x7efe940037f0 2024-05-21T17:23:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.060+0000 7efea659c640 10 monclient: get_auth_request con 0x7efe9403a480 auth_method 0 2024-05-21T17:23:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.064+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7efea80a1f60 con 0x7efe940037f0 2024-05-21T17:23:37.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.064+0000 7efe8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.064+0000 7efe8f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:37.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.292+0000 7efead1e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-05-21T17:23:37.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.292+0000 7efead1e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-05-21T17:23:37.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.292+0000 7efead1e8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:37.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.292+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 --> v1:172.21.15.3:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efe88005380 con 0x7efe940037f0 2024-05-21T17:23:37.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.292+0000 7efe8f7fe640 1 -- 172.21.15.3:0/2966528650 <== mon.8 v1:172.21.15.3:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2714 (unknown 254746657 0 505088739) 0x7efe9c014a10 con 0x7efe940037f0 2024-05-21T17:23:37.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.292+0000 7efe8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-05-21T17:23:37.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.292+0000 7efe8f7fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:37.298 INFO:teuthology.orchestra.run.smithi003.stdout:{"election_epoch":20,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","modified":"2024-05-21T17:20:49.979435Z","created":"2024-05-21T17:20:49.979435Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6790","nonce":0}]},"addr":"172.21.15.3:6790/0","public_addr":"172.21.15.3:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6790","nonce":0}]},"addr":"172.21.15.203:6790/0","public_addr":"172.21.15.203:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6791","nonce":0}]},"addr":"172.21.15.3:6791/0","public_addr":"172.21.15.3:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6791","nonce":0}]},"addr":"172.21.15.203:6791/0","public_addr":"172.21.15.203:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6792","nonce":0}]},"addr":"172.21.15.3:6792/0","public_addr":"172.21.15.3:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6792","nonce":0}]},"addr":"172.21.15.203:6792/0","public_addr":"172.21.15.203:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6793","nonce":0}]},"addr":"172.21.15.3:6793/0","public_addr":"172.21.15.3:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-21T17:23:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.296+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 >> v1:172.21.15.203:6812/343827626 conn(0x7efe9403a480 legacy=0x7efe9403c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.296+0000 7efead1e8640 10 monclient: shutdown 2024-05-21T17:23:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.296+0000 7efead1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.296+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 >> v1:172.21.15.3:6793/0 conn(0x7efe940037f0 legacy=0x7efea80a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.296+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 shutdown_connections 2024-05-21T17:23:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.296+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 >> 172.21.15.3:0/2966528650 conn(0x7efea806c8d0 msgr2=0x7efea812fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.296+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 shutdown_connections 2024-05-21T17:23:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.296+0000 7efead1e8640 1 -- 172.21.15.3:0/2966528650 wait complete. 2024-05-21T17:23:37.316 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-05-21T17:23:37.316 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-05-21T17:23:37.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-21T17:23:37.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient: get_monmap_and_config 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient: monmap: 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:37.429406+0000 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:37.429406+0000 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.428 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.429 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.429 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.429 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.429 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.429 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.429 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.429 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.429 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 1 Processor -- start 2024-05-21T17:23:37.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 1 -- start start 2024-05-21T17:23:37.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient: init 2024-05-21T17:23:37.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient: _add_conns ranks=[2,4,1,5,6,8,0,3,7] 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fdf640734d0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fdf640a0c20 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient(hunting): picked mon.noname-f con 0x7fdf64070e70 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf640a20d0 con 0x7fdf64070e70 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf640a23b0 con 0x7fdf640734d0 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf640a3430 con 0x7fdf640a0c20 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf69c4a640 10 monclient(hunting): authenticate will time out at 1084.587002s 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf637fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fdf640734d0 0x7fdf640738b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48824/0 (socket says 172.21.15.3:48824) 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf62ffd640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fdf640a0c20 0x7fdf6413bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:56824/0 (socket says 172.21.15.3:56824) 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf637fe640 1 -- 172.21.15.3:0/849515925 learned_addr learned my addr 172.21.15.3:0/849515925 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301079878 0 0) 0x7fdf640a3430 con 0x7fdf640a0c20 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient(hunting): my global_id is 4585 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf4c003530 con 0x7fdf640a0c20 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1248742165 0 0) 0x7fdf4c003530 con 0x7fdf640a0c20 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 >> v1:172.21.15.3:6790/0 conn(0x7fdf640734d0 legacy=0x7fdf640738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 >> v1:172.21.15.203:6789/0 conn(0x7fdf64070e70 legacy=0x7fdf6413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 monclient: found mon.noname-c 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:23:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64073fc0 con 0x7fdf640a0c20 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fdf54001840 con 0x7fdf640a0c20 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: dump: 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.436 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf54001a60 con 0x7fdf640a0c20 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 1 -- 172.21.15.3:0/849515925 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fdf54004b30 con 0x7fdf640a0c20 2024-05-21T17:23:37.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.424+0000 7fdf627fc640 10 monclient: dump: 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 5 monclient: authenticate success, global_id 4585 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 4 set_mon_vals no callback set 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient: shutdown 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- 172.21.15.3:0/849515925 >> v1:172.21.15.3:6791/0 conn(0x7fdf640a0c20 legacy=0x7fdf6413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- 172.21.15.3:0/849515925 shutdown_connections 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- 172.21.15.3:0/849515925 >> 172.21.15.3:0/849515925 conn(0x7fdf640a9640 msgr2=0x7fdf640a5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- 172.21.15.3:0/849515925 shutdown_connections 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- 172.21.15.3:0/849515925 wait complete. 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient: monmap: 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.440 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 Processor -- start 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- start start 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient: init 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient: _add_conns ranks=[3,8,5,1,4,0,7,6,2] 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient(hunting): picked mon.noname-g con 0x7fdf64070e70 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient(hunting): picked mon.noname-e con 0x7fdf640734d0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:37.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient(hunting): picked mon.noname-h con 0x7fdf6413fd70 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:37.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6413e2e0 con 0x7fdf64070e70 2024-05-21T17:23:37.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6413f670 con 0x7fdf6413fd70 2024-05-21T17:23:37.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6413f950 con 0x7fdf640734d0 2024-05-21T17:23:37.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf69c4a640 10 monclient(hunting): authenticate will time out at 1084.590566s 2024-05-21T17:23:37.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf62ffd640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fdf640734d0 0x7fdf6406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54640/0 (socket says 172.21.15.3:54640) 2024-05-21T17:23:37.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.428+0000 7fdf62ffd640 1 -- 172.21.15.3:0/3572088786 learned_addr learned my addr 172.21.15.3:0/3572088786 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3476907584 0 0) 0x7fdf6413f950 con 0x7fdf640734d0 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): my global_id is 4571 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf50003530 con 0x7fdf640734d0 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1200650684 0 0) 0x7fdf50003530 con 0x7fdf640734d0 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf6413f950 con 0x7fdf640734d0 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fdf54001c10 con 0x7fdf640734d0 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:37.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): dump: 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.447 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3840427145 0 0) 0x7fdf6413f950 con 0x7fdf640734d0 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 >> v1:172.21.15.203:6791/0 conn(0x7fdf6413fd70 legacy=0x7fdf641d59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 >> v1:172.21.15.203:6790/0 conn(0x7fdf64070e70 legacy=0x7fdf6406c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 monclient: found mon.e 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64139650 con 0x7fdf640734d0 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 5 monclient: authenticate success, global_id 4571 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 10 monclient: _renew_subs 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6406e100 con 0x7fdf640734d0 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 10 monclient: _renew_subs 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf6413f0b0 con 0x7fdf640734d0 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf540011c0 con 0x7fdf640734d0 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fdf540019c0 con 0x7fdf640734d0 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf61ffb640 4 set_mon_vals no callback set 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: dump: 2024-05-21T17:23:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.450 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fdf54010100 con 0x7fdf640734d0 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdf440065d0 con 0x7fdf44002870 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fdf54045ce0 con 0x7fdf640734d0 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121279883 0 0) 0x7fdf440065d0 con 0x7fdf44002870 2024-05-21T17:23:37.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient(hunting): my global_id is 4635 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf6413f950 con 0x7fdf44002870 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf63fff640 10 monclient: get_auth_request con 0x7fdf50038c60 auth_method 0 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2973191402 0 0) 0x7fdf6413f950 con 0x7fdf44002870 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf440065d0 con 0x7fdf44002870 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fdf58001820 con 0x7fdf44002870 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: dump: 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.453 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3359175982 0 0) 0x7fdf440065d0 con 0x7fdf44002870 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf4400a170 con 0x7fdf44002870 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdf58001c20 con 0x7fdf44002870 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 >> v1:172.21.15.3:6789/0 conn(0x7fdf44002870 legacy=0x7fdf44004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.440+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 >> v1:172.21.15.3:6789/0 conn(0x7fdf44002870 legacy=0x7fdf44004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdf44009e60 con 0x7fdf44009120 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955181007 0 0) 0x7fdf44009e60 con 0x7fdf44009120 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient(hunting): my global_id is 4644 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf440065d0 con 0x7fdf44009120 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1918380590 0 0) 0x7fdf440065d0 con 0x7fdf44009120 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf44009e60 con 0x7fdf44009120 2024-05-21T17:23:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fdf58004de0 con 0x7fdf44009120 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient: dump: 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1291609282 0 0) 0x7fdf44009e60 con 0x7fdf44009120 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 --> v1:172.21.15.3:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf4400c400 con 0x7fdf44009120 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 <== mon.0 v1:172.21.15.3:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7707 (unknown 0 0 3241239193) 0x7fdf580058e0 con 0x7fdf44009120 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 >> v1:172.21.15.3:6789/0 conn(0x7fdf44009120 legacy=0x7fdf440095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf4bfff640 1 -- 172.21.15.3:0/3572088786 >> v1:172.21.15.3:6789/0 conn(0x7fdf44009120 legacy=0x7fdf440095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "leader", 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 141383, 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 0, 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:37.460 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:37.461 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:37.463 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.464 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.465 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.467 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.469 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:37.474 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:37.475 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.475 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.475 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:37.475 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:37.475 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:37.479 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:23:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:37.482 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 >> v1:172.21.15.203:6812/343827626 conn(0x7fdf50038c60 legacy=0x7fdf5003b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 10 monclient: shutdown 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 >> v1:172.21.15.3:6793/0 conn(0x7fdf640734d0 legacy=0x7fdf6406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf63fff640 1 -- 172.21.15.3:0/3572088786 reap_dead start 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf63fff640 1 -- 172.21.15.3:0/3572088786 reap_dead start 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 shutdown_connections 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 >> 172.21.15.3:0/3572088786 conn(0x7fdf640a9640 msgr2=0x7fdf6409ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 shutdown_connections 2024-05-21T17:23:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.448+0000 7fdf69c4a640 1 -- 172.21.15.3:0/3572088786 wait complete. 2024-05-21T17:23:37.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:37.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-05-21T17:23:37.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient: get_monmap_and_config 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient: monmap: 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:37.600477+0000 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:37.600477+0000 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.599 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.600 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 Processor -- start 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- start start 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient: init 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient: _add_conns ranks=[0,4,8,1,2,3,7,6,5] 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient(hunting): picked mon.noname-a con 0x7fd7200a5bd0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient(hunting): picked mon.noname-c con 0x7fd7200a66c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient(hunting): picked mon.noname-e con 0x7fd7200a70c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72009e480 con 0x7fd7200a5bd0 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72009cce0 con 0x7fd7200a66c0 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72009cfc0 con 0x7fd7200a70c0 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient(hunting): authenticate will time out at 1084.757971s 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71ed9d640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fd7200a70c0 0x7fd720141370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54654/0 (socket says 172.21.15.3:54654) 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71ed9d640 1 -- 172.21.15.3:0/3332826961 learned_addr learned my addr 172.21.15.3:0/3332826961 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71e59c640 1 --1- 172.21.15.3:0/3332826961 >> v1:172.21.15.3:6789/0 conn(0x7fd7200a5bd0 0x7fd7200a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55066/0 (socket says 172.21.15.3:55066) 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332415834 0 0) 0x7fd72009cfc0 con 0x7fd7200a70c0 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): my global_id is 4580 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd70c003530 con 0x7fd7200a70c0 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763125807 0 0) 0x7fd72009e480 con 0x7fd7200a5bd0 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): my global_id is 4653 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd72009cfc0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946456622 0 0) 0x7fd72009cce0 con 0x7fd7200a66c0 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient(hunting): my global_id is 4594 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd72009e480 con 0x7fd7200a66c0 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2909356866 0 0) 0x7fd72009cfc0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 >> v1:172.21.15.3:6793/0 conn(0x7fd7200a70c0 legacy=0x7fd720141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 >> v1:172.21.15.3:6791/0 conn(0x7fd7200a66c0 legacy=0x7fd72013dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 monclient: found mon.noname-a 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7200742e0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd710001820 con 0x7fd7200a5bd0 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: dump: 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.612 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 5 monclient: authenticate success, global_id 4653 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7100019c0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 1 -- 172.21.15.3:0/3332826961 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd710004b30 con 0x7fd7200a5bd0 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 4 set_mon_vals no callback set 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd71d59a640 10 monclient: dump: 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.614 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 10 monclient: shutdown 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- 172.21.15.3:0/3332826961 >> v1:172.21.15.3:6789/0 conn(0x7fd7200a5bd0 legacy=0x7fd7200a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- 172.21.15.3:0/3332826961 shutdown_connections 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- 172.21.15.3:0/3332826961 >> 172.21.15.3:0/3332826961 conn(0x7fd7200ab7d0 msgr2=0x7fd7200a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- 172.21.15.3:0/3332826961 shutdown_connections 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.596+0000 7fd724c30640 1 -- 172.21.15.3:0/3332826961 wait complete. 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: monmap: 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 1 Processor -- start 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 1 -- start start 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: init 2024-05-21T17:23:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: _add_conns ranks=[8,2,7,6,1,3,0,5,4] 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient(hunting): picked mon.noname-e con 0x7fd7200a5bd0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient(hunting): picked mon.noname-b con 0x7fd7200a66c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient(hunting): picked mon.noname-i con 0x7fd7200a70c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7200a79f0 con 0x7fd7200a66c0 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72013fb50 con 0x7fd7200a70c0 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72013d7f0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient(hunting): authenticate will time out at 1084.761268s 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd71dd9b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fd7200a66c0 0x7fd72013c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48838/0 (socket says 172.21.15.3:48838) 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd71dd9b640 1 -- 172.21.15.3:0/328707821 learned_addr learned my addr 172.21.15.3:0/328707821 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3504723706 0 0) 0x7fd708001be0 con 0x7fd7200a66c0 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): my global_id is 134102 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd700003530 con 0x7fd7200a66c0 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 11214784 0 0) 0x7fd72013d7f0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): my global_id is 4589 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd708001be0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3999331092 0 0) 0x7fd700003530 con 0x7fd7200a66c0 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd72013d7f0 con 0x7fd7200a66c0 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd708001700 con 0x7fd7200a66c0 2024-05-21T17:23:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2540637854 0 0) 0x7fd708001be0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd700003530 con 0x7fd7200a5bd0 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd7100017e0 con 0x7fd7200a5bd0 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846842838 0 0) 0x7fd72013fb50 con 0x7fd7200a70c0 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): my global_id is 4660 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd708001be0 con 0x7fd7200a70c0 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1691657204 0 0) 0x7fd72013d7f0 con 0x7fd7200a66c0 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 >> v1:172.21.15.3:6793/0 conn(0x7fd7200a5bd0 legacy=0x7fd72013bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 >> v1:172.21.15.203:6792/0 conn(0x7fd7200a70c0 legacy=0x7fd7201d5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 monclient: found mon.b 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72013ca80 con 0x7fd7200a66c0 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 5 monclient: authenticate success, global_id 134102 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: _renew_subs 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd72013f2b0 con 0x7fd7200a66c0 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: _renew_subs 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd72013e550 con 0x7fd7200a66c0 2024-05-21T17:23:37.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7080038b0 con 0x7fd7200a66c0 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd708003de0 con 0x7fd7200a66c0 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd71cd99640 4 set_mon_vals no callback set 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: dump: 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.624 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd7040065d0 con 0x7fd704002870 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fd708003f80 con 0x7fd7200a66c0 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fd708045d10 con 0x7fd7200a66c0 2024-05-21T17:23:37.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3845236064 0 0) 0x7fd7040065d0 con 0x7fd704002870 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient(hunting): my global_id is 134111 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd72013d7f0 con 0x7fd704002870 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 570104826 0 0) 0x7fd72013d7f0 con 0x7fd704002870 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7040065d0 con 0x7fd704002870 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd710004df0 con 0x7fd704002870 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd716ffd640 10 monclient: dump: 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.626 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.627 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.600+0000 7fd71ed9d640 10 monclient: get_auth_request con 0x7fd700039370 auth_method 0 2024-05-21T17:23:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.604+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 389648328 0 0) 0x7fd7040065d0 con 0x7fd704002870 2024-05-21T17:23:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.604+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd70400a170 con 0x7fd704002870 2024-05-21T17:23:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.604+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd710007480 con 0x7fd704002870 2024-05-21T17:23:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.604+0000 7fd716ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.604+0000 7fd716ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.604+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 >> v1:172.21.15.3:6790/0 conn(0x7fd704002870 legacy=0x7fd704004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.604+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 >> v1:172.21.15.3:6790/0 conn(0x7fd704002870 legacy=0x7fd704004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:37.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd724c30640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd704009e80 con 0x7fd704009140 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3685067453 0 0) 0x7fd704009e80 con 0x7fd704009140 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient(hunting): my global_id is 134120 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7040065d0 con 0x7fd704009140 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 562152034 0 0) 0x7fd7040065d0 con 0x7fd704009140 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd704009e80 con 0x7fd704009140 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fd7100053d0 con 0x7fd704009140 2024-05-21T17:23:37.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient: dump: 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1921696203 0 0) 0x7fd704009e80 con 0x7fd704009140 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 --> v1:172.21.15.3:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd70400c400 con 0x7fd704009140 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 <== mon.2 v1:172.21.15.3:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 499127409) 0x7fd710006860 con 0x7fd704009140 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 >> v1:172.21.15.3:6790/0 conn(0x7fd704009140 legacy=0x7fd7040095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.608+0000 7fd716ffd640 1 -- 172.21.15.3:0/328707821 >> v1:172.21.15.3:6790/0 conn(0x7fd704009140 legacy=0x7fd7040095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 5655, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:37.633 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 0, 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.634 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.635 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:37.636 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.637 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.640 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:37.641 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:37.642 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:37.643 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.645 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:37.647 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.648 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.650 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:37.651 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:23:37.652 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.652 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.652 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:37.652 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:37.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 >> v1:172.21.15.203:6812/343827626 conn(0x7fd700039370 legacy=0x7fd70003b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd724c30640 10 monclient: shutdown 2024-05-21T17:23:37.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd724c30640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 >> v1:172.21.15.3:6790/0 conn(0x7fd7200a66c0 legacy=0x7fd72013c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd71ed9d640 1 -- 172.21.15.3:0/328707821 reap_dead start 2024-05-21T17:23:37.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd71ed9d640 1 -- 172.21.15.3:0/328707821 reap_dead start 2024-05-21T17:23:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 shutdown_connections 2024-05-21T17:23:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 >> 172.21.15.3:0/328707821 conn(0x7fd7200ab7d0 msgr2=0x7fd720139b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 shutdown_connections 2024-05-21T17:23:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.612+0000 7fd724c30640 1 -- 172.21.15.3:0/328707821 wait complete. 2024-05-21T17:23:37.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:37.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-05-21T17:23:37.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-21T17:23:37.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.764+0000 7ff2d552a640 10 monclient: get_monmap_and_config 2024-05-21T17:23:37.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.764+0000 7ff2d552a640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.764+0000 7ff2d552a640 10 monclient: monmap: 2024-05-21T17:23:37.770 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:37.771750+0000 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:37.771750+0000 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.772 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.772 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 Processor -- start 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- start start 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient: init 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient: _add_conns ranks=[3,5,4,2,7,8,6,1,0] 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient(hunting): picked mon.noname-g con 0x7ff2d01388b0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient(hunting): picked mon.noname-h con 0x7ff2d01393a0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient(hunting): picked mon.noname-c con 0x7ff2d009db30 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d009e720 con 0x7ff2d01388b0 2024-05-21T17:23:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d00a41b0 con 0x7ff2d009db30 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d00a43f0 con 0x7ff2d01393a0 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient(hunting): authenticate will time out at 1084.929280s 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cf7fe640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7ff2d009db30 0x7ff2d0141580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:56848/0 (socket says 172.21.15.3:56848) 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cf7fe640 1 -- 172.21.15.3:0/3275153054 learned_addr learned my addr 172.21.15.3:0/3275153054 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 128939040 0 0) 0x7ff2d00a41b0 con 0x7ff2d009db30 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient(hunting): my global_id is 4603 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2bc003530 con 0x7ff2d009db30 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2571188853 0 0) 0x7ff2bc003530 con 0x7ff2d009db30 2024-05-21T17:23:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 >> v1:172.21.15.203:6791/0 conn(0x7ff2d01393a0 legacy=0x7ff2d013de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 >> v1:172.21.15.203:6790/0 conn(0x7ff2d01388b0 legacy=0x7ff2d0138c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 monclient: found mon.noname-c 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d0069140 con 0x7ff2d009db30 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff2c8001840 con 0x7ff2d009db30 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: dump: 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.783 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 5 monclient: authenticate success, global_id 4603 2024-05-21T17:23:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2c8001a60 con 0x7ff2d009db30 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 1 -- 172.21.15.3:0/3275153054 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff2c8004bf0 con 0x7ff2d009db30 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 4 set_mon_vals no callback set 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2cdffb640 10 monclient: dump: 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient: shutdown 2024-05-21T17:23:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- 172.21.15.3:0/3275153054 >> v1:172.21.15.3:6791/0 conn(0x7ff2d009db30 legacy=0x7ff2d0141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- 172.21.15.3:0/3275153054 shutdown_connections 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- 172.21.15.3:0/3275153054 >> 172.21.15.3:0/3275153054 conn(0x7ff2d006c8d0 msgr2=0x7ff2d006cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- 172.21.15.3:0/3275153054 shutdown_connections 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 -- 172.21.15.3:0/3275153054 wait complete. 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 10 monclient: monmap: 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.768+0000 7ff2d552a640 1 Processor -- start 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 1 -- start start 2024-05-21T17:23:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient: init 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient: _add_conns ranks=[7,4,8,3,5,1,2,6,0] 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient(hunting): picked mon.noname-i con 0x7ff2d009db30 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient(hunting): picked mon.noname-c con 0x7ff2d01388b0 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient(hunting): picked mon.noname-e con 0x7ff2d01393a0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d0137850 con 0x7ff2d01388b0 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d013a240 con 0x7ff2d009db30 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d013a520 con 0x7ff2d01393a0 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient(hunting): authenticate will time out at 1084.932530s 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2ce7fc640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7ff2d01388b0 0x7ff2d01365a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:56864/0 (socket says 172.21.15.3:56864) 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2ce7fc640 1 -- 172.21.15.3:0/631485858 learned_addr learned my addr 172.21.15.3:0/631485858 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097470308 0 0) 0x7ff2d0137850 con 0x7ff2d01388b0 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): my global_id is 4612 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2b0003530 con 0x7ff2d01388b0 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962710602 0 0) 0x7ff2d013a520 con 0x7ff2d01393a0 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): my global_id is 4598 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d0137850 con 0x7ff2d01393a0 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2876428205 0 0) 0x7ff2b0003530 con 0x7ff2d01388b0 2024-05-21T17:23:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2d013a520 con 0x7ff2d01388b0 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff2b8001aa0 con 0x7ff2d01388b0 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): dump: 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1405410448 0 0) 0x7ff2d0137850 con 0x7ff2d01393a0 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2b0003530 con 0x7ff2d01393a0 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff2c8004b30 con 0x7ff2d01393a0 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:37.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): dump: 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.792 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096558904 0 0) 0x7ff2d013a240 con 0x7ff2d009db30 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): my global_id is 4669 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d0137850 con 0x7ff2d009db30 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 629581026 0 0) 0x7ff2d013a520 con 0x7ff2d01388b0 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 >> v1:172.21.15.3:6793/0 conn(0x7ff2d01393a0 legacy=0x7ff2d01d5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 >> v1:172.21.15.203:6792/0 conn(0x7ff2d009db30 legacy=0x7ff2d0135e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 monclient: found mon.c 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d0139e20 con 0x7ff2d01388b0 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 5 monclient: authenticate success, global_id 4612 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient: _renew_subs 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d0139850 con 0x7ff2d01388b0 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient: _renew_subs 2024-05-21T17:23:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2d0136cb0 con 0x7ff2d01388b0 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2b80038d0 con 0x7ff2d01388b0 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff2b8003e40 con 0x7ff2d01388b0 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: dump: 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2ccff9640 4 set_mon_vals no callback set 2024-05-21T17:23:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff2b40065d0 con 0x7ff2b4002870 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7ff2b8003fe0 con 0x7ff2d01388b0 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7ff2b8045ae0 con 0x7ff2d01388b0 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1784846094 0 0) 0x7ff2b40065d0 con 0x7ff2b4002870 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient(hunting): my global_id is 4621 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d013a520 con 0x7ff2b4002870 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3425508687 0 0) 0x7ff2d013a520 con 0x7ff2b4002870 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2b40065d0 con 0x7ff2b4002870 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff2c0001b70 con 0x7ff2b4002870 2024-05-21T17:23:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: dump: 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2cf7fe640 10 monclient: get_auth_request con 0x7ff2b00394e0 auth_method 0 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2128989930 0 0) 0x7ff2b40065d0 con 0x7ff2b4002870 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2b400a170 con 0x7ff2b4002870 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff2c0002010 con 0x7ff2b4002870 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 >> v1:172.21.15.3:6791/0 conn(0x7ff2b4002870 legacy=0x7ff2b4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.772+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 >> v1:172.21.15.3:6791/0 conn(0x7ff2b4002870 legacy=0x7ff2b4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff2b4009e60 con 0x7ff2b4009120 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3113345765 0 0) 0x7ff2b4009e60 con 0x7ff2b4009120 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient(hunting): my global_id is 4630 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2b40065d0 con 0x7ff2b4009120 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 477484138 0 0) 0x7ff2b40065d0 con 0x7ff2b4009120 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2b4009e60 con 0x7ff2b4009120 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff2c0003ff0 con 0x7ff2b4009120 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient: dump: 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3694265253 0 0) 0x7ff2b4009e60 con 0x7ff2b4009120 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 --> v1:172.21.15.3:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff2b400c400 con 0x7ff2b4009120 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 <== mon.4 v1:172.21.15.3:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7705 (unknown 0 0 4091502543) 0x7ff2c00049d0 con 0x7ff2b4009120 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 >> v1:172.21.15.3:6791/0 conn(0x7ff2b4009120 legacy=0x7ff2b40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2c77fe640 1 -- 172.21.15.3:0/631485858 >> v1:172.21.15.3:6791/0 conn(0x7ff2b4009120 legacy=0x7ff2b40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 141715, 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:37.804 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 0, 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.805 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.806 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:37.807 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:37.808 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:37.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:37.811 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.812 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:37.817 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.818 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:37.819 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.820 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.822 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:37.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.824 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:37.824 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:37.824 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 5 2024-05-21T17:23:37.824 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.824 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.824 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:37.824 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 >> v1:172.21.15.203:6812/343827626 conn(0x7ff2b00394e0 legacy=0x7ff2b003b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 10 monclient: shutdown 2024-05-21T17:23:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 >> v1:172.21.15.3:6791/0 conn(0x7ff2d01388b0 legacy=0x7ff2d01365a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2cf7fe640 1 -- 172.21.15.3:0/631485858 reap_dead start 2024-05-21T17:23:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2cf7fe640 1 -- 172.21.15.3:0/631485858 reap_dead start 2024-05-21T17:23:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 shutdown_connections 2024-05-21T17:23:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 >> 172.21.15.3:0/631485858 conn(0x7ff2d006c8d0 msgr2=0x7ff2d00a6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 shutdown_connections 2024-05-21T17:23:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.780+0000 7ff2d552a640 1 -- 172.21.15.3:0/631485858 wait complete. 2024-05-21T17:23:37.826 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:37.826 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-05-21T17:23:37.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-05-21T17:23:37.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient: get_monmap_and_config 2024-05-21T17:23:37.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient: monmap: 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:37.945080+0000 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:37.945080+0000 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 1 Processor -- start 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 1 -- start start 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient: init 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient: _add_conns ranks=[8,7,2,6,1,5,0,3,4] 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient(hunting): picked mon.noname-e con 0x7fbdac0a5bd0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient(hunting): picked mon.noname-i con 0x7fbdac0a66c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient(hunting): picked mon.noname-b con 0x7fbdac0a70c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdac09e480 con 0x7fbdac0a70c0 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdac09cce0 con 0x7fbdac0a66c0 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdac09cfc0 con 0x7fbdac0a5bd0 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient(hunting): authenticate will time out at 1085.102507s 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb2485640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fbdac0a70c0 0x7fbdac141370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48864/0 (socket says 172.21.15.3:48864) 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb2485640 1 -- 172.21.15.3:0/48388346 learned_addr learned my addr 172.21.15.3:0/48388346 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb1c84640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fbdac0a5bd0 0x7fbdac0a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54678/0 (socket says 172.21.15.3:54678) 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3362201287 0 0) 0x7fbdac09e480 con 0x7fbdac0a70c0 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): my global_id is 134129 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd94003530 con 0x7fbdac0a70c0 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223480424 0 0) 0x7fbdac09cfc0 con 0x7fbdac0a5bd0 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): my global_id is 4607 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdac09e480 con 0x7fbdac0a5bd0 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1618437067 0 0) 0x7fbdac09cce0 con 0x7fbdac0a66c0 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient(hunting): my global_id is 4678 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdac09cfc0 con 0x7fbdac0a66c0 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1374054334 0 0) 0x7fbd94003530 con 0x7fbdac0a70c0 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 >> v1:172.21.15.3:6793/0 conn(0x7fbdac0a5bd0 legacy=0x7fbdac0a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 >> v1:172.21.15.203:6792/0 conn(0x7fbdac0a66c0 legacy=0x7fbdac13dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 monclient: found mon.noname-b 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdac0742e0 con 0x7fbdac0a70c0 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbd9c001840 con 0x7fbdac0a70c0 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 5 monclient: authenticate success, global_id 134129 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: dump: 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd9c001a60 con 0x7fbdac0a70c0 2024-05-21T17:23:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 1 -- 172.21.15.3:0/48388346 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbd9c001d50 con 0x7fbdac0a70c0 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb0c82640 10 monclient: dump: 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.940+0000 7fbdb3ee8640 4 set_mon_vals no callback set 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: shutdown 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/48388346 >> v1:172.21.15.3:6790/0 conn(0x7fbdac0a70c0 legacy=0x7fbdac141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/48388346 shutdown_connections 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/48388346 >> 172.21.15.3:0/48388346 conn(0x7fbdac0ab7d0 msgr2=0x7fbdac0a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/48388346 shutdown_connections 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/48388346 wait complete. 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: build_initial_monmap 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: monmap: 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 Processor -- start 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- start start 2024-05-21T17:23:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: init 2024-05-21T17:23:37.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:37.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: _add_conns ranks=[3,8,4,5,7,1,2,0,6] 2024-05-21T17:23:37.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient(hunting): picked mon.noname-g con 0x7fbdac0a5bd0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:37.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient(hunting): picked mon.noname-e con 0x7fbdac0a66c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:37.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient(hunting): picked mon.noname-c con 0x7fbdac0a70c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:37.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdac123c10 con 0x7fbdac0a5bd0 2024-05-21T17:23:37.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdac0a79f0 con 0x7fbdac0a70c0 2024-05-21T17:23:37.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdac121990 con 0x7fbdac0a66c0 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient(hunting): authenticate will time out at 1085.105953s 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb1483640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fbdac0a66c0 0x7fbdac1d2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54682/0 (socket says 172.21.15.3:54682) 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb1483640 1 -- 172.21.15.3:0/2654682388 learned_addr learned my addr 172.21.15.3:0/2654682388 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107186653 0 0) 0x7fbdac121990 con 0x7fbdac0a66c0 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): my global_id is 4616 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd98003530 con 0x7fbdac0a66c0 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762341578 0 0) 0x7fbdac0a79f0 con 0x7fbdac0a70c0 2024-05-21T17:23:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): my global_id is 4639 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdac121990 con 0x7fbdac0a70c0 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1540010852 0 0) 0x7fbdac121990 con 0x7fbdac0a70c0 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdac0a79f0 con 0x7fbdac0a70c0 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbd9c001c10 con 0x7fbdac0a70c0 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): dump: 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.967 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2403791091 0 0) 0x7fbd98003530 con 0x7fbdac0a66c0 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdac121990 con 0x7fbdac0a66c0 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432250491 0 0) 0x7fbdac123c10 con 0x7fbdac0a5bd0 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): my global_id is 4611 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd98003530 con 0x7fbdac0a5bd0 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbda0001ec0 con 0x7fbdac0a66c0 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient(hunting): dump: 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3534180811 0 0) 0x7fbdac0a79f0 con 0x7fbdac0a70c0 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 >> v1:172.21.15.3:6793/0 conn(0x7fbdac0a66c0 legacy=0x7fbdac1d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 >> v1:172.21.15.203:6790/0 conn(0x7fbdac0a5bd0 legacy=0x7fbdac1cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 monclient: found mon.c 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdac123930 con 0x7fbdac0a70c0 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 5 monclient: authenticate success, global_id 4639 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: _renew_subs 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbdac13c7c0 con 0x7fbdac0a70c0 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: _renew_subs 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fbdac141ef0 con 0x7fbdac0a70c0 2024-05-21T17:23:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd9c001180 con 0x7fbdac0a70c0 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbd9c001920 con 0x7fbdac0a70c0 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdabfff640 4 set_mon_vals no callback set 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdaa7fc640 10 monclient: dump: 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.972 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.944+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbd8c0065d0 con 0x7fbd8c002870 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fbd9c010100 con 0x7fbdac0a70c0 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fbd9c045b10 con 0x7fbdac0a70c0 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2078044371 0 0) 0x7fbd8c0065d0 con 0x7fbd8c002870 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient(hunting): my global_id is 4470 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdac0a79f0 con 0x7fbd8c002870 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 907330438 0 0) 0x7fbdac0a79f0 con 0x7fbd8c002870 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd8c0065d0 con 0x7fbd8c002870 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdb1483640 10 monclient: get_auth_request con 0x7fbd98039430 auth_method 0 2024-05-21T17:23:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbda4001350 con 0x7fbd8c002870 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient: dump: 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.976 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.976 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.976 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.976 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.976 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 506778057 0 0) 0x7fbd8c0065d0 con 0x7fbd8c002870 2024-05-21T17:23:37.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd8c00a170 con 0x7fbd8c002870 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbda40016b0 con 0x7fbd8c002870 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 >> v1:172.21.15.3:6792/0 conn(0x7fbd8c002870 legacy=0x7fbd8c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.948+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 >> v1:172.21.15.3:6792/0 conn(0x7fbd8c002870 legacy=0x7fbd8c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdb3ee8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbd8c009e80 con 0x7fbd8c009140 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2624139007 0 0) 0x7fbd8c009e80 con 0x7fbd8c009140 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 10 monclient(hunting): my global_id is 4479 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd8c0065d0 con 0x7fbd8c009140 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1948537478 0 0) 0x7fbd8c0065d0 con 0x7fbd8c009140 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd8c009e80 con 0x7fbd8c009140 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbda4004d20 con 0x7fbd8c009140 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.952+0000 7fbdaa7fc640 10 monclient: dump: 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:37.978 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1194760949 0 0) 0x7fbd8c009e80 con 0x7fbd8c009140 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 --> v1:172.21.15.3:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbd8c00c420 con 0x7fbd8c009140 2024-05-21T17:23:37.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 <== mon.6 v1:172.21.15.3:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 777979555) 0x7fbda4006170 con 0x7fbd8c009140 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdaa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdaa7fc640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 >> v1:172.21.15.3:6792/0 conn(0x7fbd8c009140 legacy=0x7fbd8c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdaa7fc640 1 -- 172.21.15.3:0/2654682388 >> v1:172.21.15.3:6792/0 conn(0x7fbd8c009140 legacy=0x7fbd8c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 141879, 2024-05-21T17:23:37.980 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 0, 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:37.981 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:37.984 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:37.985 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.988 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.989 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.992 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:37.993 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:37.994 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:37.995 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:37.996 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.997 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.998 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:37.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.001 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:38.001 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 >> v1:172.21.15.203:6812/343827626 conn(0x7fbd98039430 legacy=0x7fbd9803b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb3ee8640 10 monclient: shutdown 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb3ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 >> v1:172.21.15.3:6791/0 conn(0x7fbdac0a70c0 legacy=0x7fbdac1d6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb2485640 1 -- 172.21.15.3:0/2654682388 reap_dead start 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb2485640 1 -- 172.21.15.3:0/2654682388 reap_dead start 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 shutdown_connections 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 >> 172.21.15.3:0/2654682388 conn(0x7fbdac0ab7d0 msgr2=0x7fbdac075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 shutdown_connections 2024-05-21T17:23:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:37.956+0000 7fbdb3ee8640 1 -- 172.21.15.3:0/2654682388 wait complete. 2024-05-21T17:23:38.003 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:38.003 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-05-21T17:23:38.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-05-21T17:23:38.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient: get_monmap_and_config 2024-05-21T17:23:38.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient: monmap: 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:38.122181+0000 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:38.122181+0000 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.122 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.122 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.122 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.122 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 1 Processor -- start 2024-05-21T17:23:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 1 -- start start 2024-05-21T17:23:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient: init 2024-05-21T17:23:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient: _add_conns ranks=[8,5,3,4,6,1,7,0,2] 2024-05-21T17:23:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient(hunting): picked mon.noname-e con 0x7f59f00a5bd0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient(hunting): picked mon.noname-h con 0x7f59f00a66c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient(hunting): picked mon.noname-g con 0x7f59f00a70c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f009e480 con 0x7f59f00a70c0 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f009cce0 con 0x7f59f00a66c0 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f009cfc0 con 0x7f59f00a5bd0 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59f48d1640 10 monclient(hunting): authenticate will time out at 1085.279736s 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59ee59c640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f59f00a5bd0 0x7f59f00a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54688/0 (socket says 172.21.15.3:54688) 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.116+0000 7f59ee59c640 1 -- 172.21.15.3:0/2481906901 learned_addr learned my addr 172.21.15.3:0/2481906901 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1413826677 0 0) 0x7f59f009cfc0 con 0x7f59f00a5bd0 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient(hunting): my global_id is 4625 2024-05-21T17:23:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59dc003530 con 0x7f59f00a5bd0 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 720388585 0 0) 0x7f59dc003530 con 0x7f59f00a5bd0 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 >> v1:172.21.15.203:6791/0 conn(0x7f59f00a66c0 legacy=0x7f59f013dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 >> v1:172.21.15.203:6790/0 conn(0x7f59f00a70c0 legacy=0x7f59f0141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 monclient: found mon.noname-e 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f00742e0 con 0x7f59f00a5bd0 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f59e8001bc0 con 0x7f59f00a5bd0 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 5 monclient: authenticate success, global_id 4625 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: dump: 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59e8001de0 con 0x7f59f00a5bd0 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 1 -- 172.21.15.3:0/2481906901 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f59e8004f70 con 0x7f59f00a5bd0 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 4 set_mon_vals no callback set 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ed59a640 10 monclient: dump: 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: shutdown 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- 172.21.15.3:0/2481906901 >> v1:172.21.15.3:6793/0 conn(0x7f59f00a5bd0 legacy=0x7f59f00a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- 172.21.15.3:0/2481906901 shutdown_connections 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- 172.21.15.3:0/2481906901 >> 172.21.15.3:0/2481906901 conn(0x7f59f00ab7d0 msgr2=0x7f59f00a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- 172.21.15.3:0/2481906901 shutdown_connections 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- 172.21.15.3:0/2481906901 wait complete. 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: monmap: 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 Processor -- start 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- start start 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: init 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: _add_conns ranks=[5,6,1,2,0,8,3,7,4] 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient(hunting): picked mon.noname-h con 0x7f59f00a5bd0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient(hunting): picked mon.noname-d con 0x7f59f00a66c0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient(hunting): picked mon.noname-f con 0x7f59f00a70c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f01409f0 con 0x7f59f00a70c0 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f00a79f0 con 0x7f59f00a5bd0 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f013f920 con 0x7f59f00a66c0 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient(hunting): authenticate will time out at 1085.282802s 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59edd9b640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f59f00a66c0 0x7f59f01d26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:42932/0 (socket says 172.21.15.3:42932) 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59edd9b640 1 -- 172.21.15.3:0/68994824 learned_addr learned my addr 172.21.15.3:0/68994824 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220612267 0 0) 0x7f59f013f920 con 0x7f59f00a66c0 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): my global_id is 4488 2024-05-21T17:23:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d0003530 con 0x7f59f00a66c0 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 938232411 0 0) 0x7f59d0003530 con 0x7f59f00a66c0 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59f013f920 con 0x7f59f00a66c0 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f59d8001b80 con 0x7f59f00a66c0 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3247163863 0 0) 0x7f59f00a79f0 con 0x7f59f00a5bd0 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): my global_id is 4694 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d0003530 con 0x7f59f00a5bd0 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322898703 0 0) 0x7f59f009e480 con 0x7f59f00a70c0 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient(hunting): my global_id is 134101 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59f00a79f0 con 0x7f59f00a70c0 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3015324974 0 0) 0x7f59f013f920 con 0x7f59f00a66c0 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 >> v1:172.21.15.203:6791/0 conn(0x7f59f00a5bd0 legacy=0x7f59f01cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 >> v1:172.21.15.203:6789/0 conn(0x7f59f00a70c0 legacy=0x7f59f01d5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 monclient: found mon.d 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f0141c10 con 0x7f59f00a66c0 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 5 monclient: authenticate success, global_id 4488 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: _renew_subs 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59f01cd360 con 0x7f59f00a66c0 2024-05-21T17:23:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: _renew_subs 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f59f01ce0c0 con 0x7f59f00a66c0 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59d80038d0 con 0x7f59f00a66c0 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f59d8003f00 con 0x7f59f00a66c0 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59ecd99640 4 set_mon_vals no callback set 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.120+0000 7f59e6ffd640 10 monclient: dump: 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59f48d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59d40065d0 con 0x7f59d4002870 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f59d80040a0 con 0x7f59f00a66c0 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f59d8045bc0 con 0x7f59f00a66c0 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2391531420 0 0) 0x7f59d40065d0 con 0x7f59d4002870 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient(hunting): my global_id is 4634 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59f013f920 con 0x7f59d4002870 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 350472455 0 0) 0x7f59f013f920 con 0x7f59d4002870 2024-05-21T17:23:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59d40065d0 con 0x7f59d4002870 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f59e8004fb0 con 0x7f59d4002870 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59eed9d640 10 monclient: get_auth_request con 0x7f59d0038dc0 auth_method 0 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient: dump: 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2850891328 0 0) 0x7f59d40065d0 con 0x7f59d4002870 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59d400a170 con 0x7f59d4002870 2024-05-21T17:23:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f59e8001a10 con 0x7f59d4002870 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 >> v1:172.21.15.3:6793/0 conn(0x7f59d4002870 legacy=0x7f59d4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.124+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 >> v1:172.21.15.3:6793/0 conn(0x7f59d4002870 legacy=0x7f59d4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.128+0000 7f59f48d1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.128+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59d4009e80 con 0x7f59d4009140 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.128+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392481826 0 0) 0x7f59d4009e80 con 0x7f59d4009140 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.128+0000 7f59e6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.128+0000 7f59e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.128+0000 7f59e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.128+0000 7f59e6ffd640 10 monclient(hunting): my global_id is 4643 2024-05-21T17:23:38.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.128+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d40065d0 con 0x7f59d4009140 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1960252871 0 0) 0x7f59d40065d0 con 0x7f59d4009140 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59d4009e80 con 0x7f59d4009140 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f59e8005230 con 0x7f59d4009140 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 10 monclient: dump: 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.157 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 325001185 0 0) 0x7f59d4009e80 con 0x7f59d4009140 2024-05-21T17:23:38.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 --> v1:172.21.15.3:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59d400c420 con 0x7f59d4009140 2024-05-21T17:23:38.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 <== mon.8 v1:172.21.15.3:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 3859015803) 0x7f59e80086e0 con 0x7f59d4009140 2024-05-21T17:23:38.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:38.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:38.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 >> v1:172.21.15.3:6793/0 conn(0x7f59d4009140 legacy=0x7f59d40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59e6ffd640 1 -- 172.21.15.3:0/68994824 >> v1:172.21.15.3:6793/0 conn(0x7f59d4009140 legacy=0x7f59d40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.159 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 142063, 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:38.181 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:38.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 >> v1:172.21.15.203:6812/343827626 conn(0x7f59d0038dc0 legacy=0x7f59d003b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59f48d1640 10 monclient: shutdown 2024-05-21T17:23:38.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59f48d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 >> v1:172.21.15.3:6792/0 conn(0x7f59f00a66c0 legacy=0x7f59f01d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59eed9d640 1 -- 172.21.15.3:0/68994824 reap_dead start 2024-05-21T17:23:38.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59eed9d640 1 -- 172.21.15.3:0/68994824 reap_dead start 2024-05-21T17:23:38.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 shutdown_connections 2024-05-21T17:23:38.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 >> 172.21.15.3:0/68994824 conn(0x7f59f00ab7d0 msgr2=0x7f59f0075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 shutdown_connections 2024-05-21T17:23:38.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.132+0000 7f59f48d1640 1 -- 172.21.15.3:0/68994824 wait complete. 2024-05-21T17:23:38.183 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:38.184 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-05-21T17:23:38.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient: get_monmap_and_config 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient: monmap: 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:38.308457+0000 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:38.308457+0000 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.307 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.308 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 Processor -- start 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- start start 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient: init 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient: _add_conns ranks=[1,8,7,5,0,4,6,2,3] 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient(hunting): picked mon.noname-f con 0x7fc92c0a5bd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient(hunting): picked mon.noname-e con 0x7fc92c0a66c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient(hunting): picked mon.noname-i con 0x7fc92c0a70c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c09e480 con 0x7fc92c0a5bd0 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c09cce0 con 0x7fc92c0a70c0 2024-05-21T17:23:38.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c09cfc0 con 0x7fc92c0a66c0 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient(hunting): authenticate will time out at 1085.465889s 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc929d9b640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7fc92c0a66c0 0x7fc92c13dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54712/0 (socket says 172.21.15.3:54712) 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc929d9b640 1 -- 172.21.15.3:0/2861962409 learned_addr learned my addr 172.21.15.3:0/2861962409 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3038894352 0 0) 0x7fc92c09cfc0 con 0x7fc92c0a66c0 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient(hunting): my global_id is 4652 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc914003530 con 0x7fc92c0a66c0 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4111393901 0 0) 0x7fc914003530 con 0x7fc92c0a66c0 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 >> v1:172.21.15.203:6792/0 conn(0x7fc92c0a70c0 legacy=0x7fc92c141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 >> v1:172.21.15.203:6789/0 conn(0x7fc92c0a5bd0 legacy=0x7fc92c0a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 monclient: found mon.noname-e 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c0742e0 con 0x7fc92c0a66c0 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 5 monclient: authenticate success, global_id 4652 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc91c001840 con 0x7fc92c0a66c0 2024-05-21T17:23:38.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: dump: 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.319 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc91c001b10 con 0x7fc92c0a66c0 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 1 -- 172.21.15.3:0/2861962409 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc91c004bf0 con 0x7fc92c0a66c0 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 4 set_mon_vals no callback set 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:38.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc92959a640 10 monclient: dump: 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.321 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 10 monclient: shutdown 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- 172.21.15.3:0/2861962409 >> v1:172.21.15.3:6793/0 conn(0x7fc92c0a66c0 legacy=0x7fc92c13dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- 172.21.15.3:0/2861962409 shutdown_connections 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- 172.21.15.3:0/2861962409 >> 172.21.15.3:0/2861962409 conn(0x7fc92c0ab7d0 msgr2=0x7fc92c0a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- 172.21.15.3:0/2861962409 shutdown_connections 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.304+0000 7fc930d28640 1 -- 172.21.15.3:0/2861962409 wait complete. 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: monmap: 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.323 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 1 Processor -- start 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 1 -- start start 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: init 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: _add_conns ranks=[1,8,0,5,7,6,3,2,4] 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient(hunting): picked mon.noname-f con 0x7fc92c0a5bd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient(hunting): picked mon.noname-e con 0x7fc92c0a66c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient(hunting): picked mon.noname-a con 0x7fc92c0a70c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c141a60 con 0x7fc92c0a70c0 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c141d40 con 0x7fc92c0a5bd0 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c1396b0 con 0x7fc92c0a66c0 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient(hunting): authenticate will time out at 1085.469122s 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc92ad9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc92c0a70c0 0x7fc92c1d5b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55076/0 (socket says 172.21.15.3:55076) 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc92ad9d640 1 -- 172.21.15.3:0/1350523548 learned_addr learned my addr 172.21.15.3:0/1350523548 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517433614 0 0) 0x7fc92c1396b0 con 0x7fc92c0a66c0 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): my global_id is 4661 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc90c003530 con 0x7fc92c0a66c0 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2589849629 0 0) 0x7fc92c141a60 con 0x7fc92c0a70c0 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): my global_id is 4662 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc92c1396b0 con 0x7fc92c0a70c0 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 584240259 0 0) 0x7fc90c003530 con 0x7fc92c0a66c0 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc92c141a60 con 0x7fc92c0a66c0 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc91c001800 con 0x7fc92c0a66c0 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): dump: 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.326 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3252279084 0 0) 0x7fc92c1396b0 con 0x7fc92c0a70c0 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc90c003530 con 0x7fc92c0a70c0 2024-05-21T17:23:38.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 746581937 0 0) 0x7fc92c141d40 con 0x7fc92c0a5bd0 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): my global_id is 134119 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc92c1396b0 con 0x7fc92c0a5bd0 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc924001da0 con 0x7fc92c0a70c0 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): dump: 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 274562613 0 0) 0x7fc92c141a60 con 0x7fc92c0a66c0 2024-05-21T17:23:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 >> v1:172.21.15.203:6789/0 conn(0x7fc92c0a5bd0 legacy=0x7fc92c13de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 >> v1:172.21.15.3:6789/0 conn(0x7fc92c0a70c0 legacy=0x7fc92c1d5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 monclient: found mon.e 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c140330 con 0x7fc92c0a66c0 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 5 monclient: authenticate success, global_id 4661 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: _renew_subs 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc92c13fa50 con 0x7fc92c0a66c0 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: _renew_subs 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc92c1424f0 con 0x7fc92c0a66c0 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc91c004c50 con 0x7fc92c0a66c0 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc91c001a20 con 0x7fc92c0a66c0 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc928d99640 4 set_mon_vals no callback set 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient: dump: 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc9100065d0 con 0x7fc910002870 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fc91c010140 con 0x7fc92c0a66c0 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fc91c045b50 con 0x7fc92c0a66c0 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc92ad9d640 10 monclient: get_auth_request con 0x7fc90c039540 auth_method 0 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450497980 0 0) 0x7fc9100065d0 con 0x7fc910002870 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 10 monclient(hunting): my global_id is 134128 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.308+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc92c141a60 con 0x7fc910002870 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3587359796 0 0) 0x7fc92c141a60 con 0x7fc910002870 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9100065d0 con 0x7fc910002870 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc9200018c0 con 0x7fc910002870 2024-05-21T17:23:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 10 monclient: dump: 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.334 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3719327129 0 0) 0x7fc9100065d0 con 0x7fc910002870 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc91000a170 con 0x7fc910002870 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc920001ca0 con 0x7fc910002870 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 >> v1:172.21.15.203:6789/0 conn(0x7fc910002870 legacy=0x7fc910004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.312+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 >> v1:172.21.15.203:6789/0 conn(0x7fc910002870 legacy=0x7fc910004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.316+0000 7fc930d28640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.316+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc910009e80 con 0x7fc910009140 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2574083896 0 0) 0x7fc910009e80 con 0x7fc910009140 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient(hunting): my global_id is 134137 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9100065d0 con 0x7fc910009140 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 428784609 0 0) 0x7fc9100065d0 con 0x7fc910009140 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc910009e80 con 0x7fc910009140 2024-05-21T17:23:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc920003c90 con 0x7fc910009140 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient: dump: 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4137227892 0 0) 0x7fc910009e80 con 0x7fc910009140 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc91000c420 con 0x7fc910009140 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 3335309424) 0x7fc920003c90 con 0x7fc910009140 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 >> v1:172.21.15.203:6789/0 conn(0x7fc910009140 legacy=0x7fc9100095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc91affd640 1 -- 172.21.15.3:0/1350523548 >> v1:172.21.15.203:6789/0 conn(0x7fc910009140 legacy=0x7fc9100095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 6359, 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:38.340 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:38.341 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.342 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.343 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:38.344 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.345 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.347 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.349 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.351 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:38.352 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:38.360 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 >> v1:172.21.15.203:6812/343827626 conn(0x7fc90c039540 legacy=0x7fc90c03ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc930d28640 10 monclient: shutdown 2024-05-21T17:23:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc930d28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 >> v1:172.21.15.3:6793/0 conn(0x7fc92c0a66c0 legacy=0x7fc92c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc92ad9d640 1 -- 172.21.15.3:0/1350523548 reap_dead start 2024-05-21T17:23:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc92ad9d640 1 -- 172.21.15.3:0/1350523548 reap_dead start 2024-05-21T17:23:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 shutdown_connections 2024-05-21T17:23:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 >> 172.21.15.3:0/1350523548 conn(0x7fc92c0ab7d0 msgr2=0x7fc92c0a9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 shutdown_connections 2024-05-21T17:23:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.320+0000 7fc930d28640 1 -- 172.21.15.3:0/1350523548 wait complete. 2024-05-21T17:23:38.362 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:38.362 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-05-21T17:23:38.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-05-21T17:23:38.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.480+0000 7f5b2afc4640 10 monclient: get_monmap_and_config 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.480+0000 7f5b2afc4640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.480+0000 7f5b2afc4640 10 monclient: monmap: 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:38.487348+0000 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:38.487348+0000 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.486 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.487 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.487 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.487 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.487 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.487 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.487 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.487 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.487 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 Processor -- start 2024-05-21T17:23:38.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- start start 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: init 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: _add_conns ranks=[3,8,5,2,0,7,4,6,1] 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): picked mon.noname-g con 0x7f5b240734d0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): picked mon.noname-e con 0x7f5b240a0c20 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): picked mon.noname-h con 0x7f5b24070e70 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b240a20d0 con 0x7f5b240734d0 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b240a23b0 con 0x7f5b24070e70 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b240a3430 con 0x7f5b240a0c20 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): authenticate will time out at 1085.645020s 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b23fff640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f5b240a0c20 0x7f5b2413bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54744/0 (socket says 172.21.15.3:54744) 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b23fff640 1 -- 172.21.15.3:0/2023636823 learned_addr learned my addr 172.21.15.3:0/2023636823 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069158037 0 0) 0x7f5b240a3430 con 0x7f5b240a0c20 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient(hunting): my global_id is 4670 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b10003530 con 0x7f5b240a0c20 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2824045983 0 0) 0x7f5b10003530 con 0x7f5b240a0c20 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 >> v1:172.21.15.203:6791/0 conn(0x7f5b24070e70 legacy=0x7f5b2413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 >> v1:172.21.15.203:6790/0 conn(0x7f5b240734d0 legacy=0x7f5b240738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 monclient: found mon.noname-e 2024-05-21T17:23:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b24073fc0 con 0x7f5b240a0c20 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b0c001840 con 0x7f5b240a0c20 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 5 monclient: authenticate success, global_id 4670 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: dump: 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.497 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b0c001a60 con 0x7f5b240a0c20 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 1 -- 172.21.15.3:0/2023636823 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b0c004bf0 con 0x7f5b240a0c20 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 4 set_mon_vals no callback set 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b237fe640 10 monclient: dump: 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.503 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: shutdown 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2023636823 >> v1:172.21.15.3:6793/0 conn(0x7f5b240a0c20 legacy=0x7f5b2413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2023636823 shutdown_connections 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2023636823 >> 172.21.15.3:0/2023636823 conn(0x7f5b240a9640 msgr2=0x7f5b240a5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2023636823 shutdown_connections 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2023636823 wait complete. 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: monmap: 2024-05-21T17:23:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 Processor -- start 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- start start 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: init 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient: _add_conns ranks=[7,8,4,3,0,5,1,2,6] 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): picked mon.noname-i con 0x7f5b24070e70 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): picked mon.noname-e con 0x7f5b240734d0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): picked mon.noname-c con 0x7f5b241cbeb0 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b2413fb00 con 0x7f5b241cbeb0 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b24074260 con 0x7f5b24070e70 2024-05-21T17:23:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b2413d810 con 0x7f5b240734d0 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.484+0000 7f5b2afc4640 10 monclient(hunting): authenticate will time out at 1085.647920s 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b23fff640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f5b240734d0 0x7f5b241d0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54748/0 (socket says 172.21.15.3:54748) 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b23fff640 1 -- 172.21.15.3:0/2298180997 learned_addr learned my addr 172.21.15.3:0/2298180997 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365588976 0 0) 0x7f5b2413d810 con 0x7f5b240734d0 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): my global_id is 4679 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b04003530 con 0x7f5b240734d0 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2740430911 0 0) 0x7f5b2413fb00 con 0x7f5b241cbeb0 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): my global_id is 4648 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b2413d810 con 0x7f5b241cbeb0 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2183708574 0 0) 0x7f5b04003530 con 0x7f5b240734d0 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b2413fb00 con 0x7f5b240734d0 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b0c001c10 con 0x7f5b240734d0 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): dump: 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:38.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 695206050 0 0) 0x7f5b2413d810 con 0x7f5b241cbeb0 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b04003530 con 0x7f5b241cbeb0 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1669707555 0 0) 0x7f5b24074260 con 0x7f5b24070e70 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): my global_id is 4696 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b2413d810 con 0x7f5b24070e70 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b140019e0 con 0x7f5b241cbeb0 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): dump: 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.510 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.511 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 673441743 0 0) 0x7f5b2413fb00 con 0x7f5b240734d0 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 >> v1:172.21.15.203:6792/0 conn(0x7f5b24070e70 legacy=0x7f5b241cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 >> v1:172.21.15.3:6791/0 conn(0x7f5b241cbeb0 legacy=0x7f5b241d3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 monclient: found mon.e 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2413f260 con 0x7f5b240734d0 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 5 monclient: authenticate success, global_id 4679 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 10 monclient: _renew_subs 2024-05-21T17:23:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2413f540 con 0x7f5b240734d0 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 10 monclient: _renew_subs 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f5b241cb3a0 con 0x7f5b240734d0 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b0c0011c0 con 0x7f5b240734d0 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b0c001920 con 0x7f5b240734d0 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b22ffd640 4 set_mon_vals no callback set 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: dump: 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.513 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.514 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5b080065d0 con 0x7f5b08002870 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f5b0c010100 con 0x7f5b240734d0 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f5b0c045c10 con 0x7f5b240734d0 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b29561640 10 monclient: get_auth_request con 0x7f5b040393e0 auth_method 0 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3357666473 0 0) 0x7f5b080065d0 con 0x7f5b08002870 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient(hunting): my global_id is 4638 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b2413fb00 con 0x7f5b08002870 2024-05-21T17:23:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2523366853 0 0) 0x7f5b2413fb00 con 0x7f5b08002870 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b080065d0 con 0x7f5b08002870 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b1c001c70 con 0x7f5b08002870 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 10 monclient: dump: 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3643397663 0 0) 0x7f5b080065d0 con 0x7f5b08002870 2024-05-21T17:23:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.488+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b0800a170 con 0x7f5b08002870 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.492+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5b1c001f60 con 0x7f5b08002870 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.492+0000 7f5b217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.492+0000 7f5b217fa640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.492+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 >> v1:172.21.15.203:6790/0 conn(0x7f5b08002870 legacy=0x7f5b08004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.492+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 >> v1:172.21.15.203:6790/0 conn(0x7f5b08002870 legacy=0x7f5b08004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b2afc4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5b08009e80 con 0x7f5b08009140 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024397458 0 0) 0x7f5b08009e80 con 0x7f5b08009140 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 10 monclient(hunting): my global_id is 4647 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b080065d0 con 0x7f5b08009140 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4248167436 0 0) 0x7f5b080065d0 con 0x7f5b08009140 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b08009e80 con 0x7f5b08009140 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5b1c004010 con 0x7f5b08009140 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 10 monclient: dump: 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1809291925 0 0) 0x7f5b08009e80 con 0x7f5b08009140 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.496+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 --> v1:172.21.15.203:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b0800c420 con 0x7f5b08009140 2024-05-21T17:23:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 <== mon.3 v1:172.21.15.203:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 2220653781) 0x7f5b1c0049d0 con 0x7f5b08009140 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b217fa640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 >> v1:172.21.15.203:6790/0 conn(0x7f5b08009140 legacy=0x7f5b080095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b217fa640 1 -- 172.21.15.3:0/2298180997 >> v1:172.21.15.203:6790/0 conn(0x7f5b08009140 legacy=0x7f5b080095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:23:38.522 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 142403, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:23:38.523 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.524 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.525 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.526 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.527 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:38.528 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.529 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.530 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.533 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:38.534 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:38.535 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.536 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.537 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.539 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.540 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.541 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:38.542 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 >> v1:172.21.15.203:6812/343827626 conn(0x7f5b040393e0 legacy=0x7f5b0403b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b2afc4640 10 monclient: shutdown 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b2afc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 >> v1:172.21.15.3:6793/0 conn(0x7f5b240734d0 legacy=0x7f5b241d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b29561640 1 -- 172.21.15.3:0/2298180997 reap_dead start 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b29561640 1 -- 172.21.15.3:0/2298180997 reap_dead start 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 shutdown_connections 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 >> 172.21.15.3:0/2298180997 conn(0x7f5b240a9640 msgr2=0x7f5b240a5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 shutdown_connections 2024-05-21T17:23:38.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.500+0000 7f5b2afc4640 1 -- 172.21.15.3:0/2298180997 wait complete. 2024-05-21T17:23:38.545 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:38.545 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-05-21T17:23:38.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-05-21T17:23:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient: get_monmap_and_config 2024-05-21T17:23:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient: monmap: 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:38.657078+0000 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:38.657078+0000 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.657 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 Processor -- start 2024-05-21T17:23:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- start start 2024-05-21T17:23:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient: init 2024-05-21T17:23:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient: _add_conns ranks=[2,7,8,4,5,0,6,1,3] 2024-05-21T17:23:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient(hunting): picked mon.noname-b con 0x7f48f00a1d00 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient(hunting): picked mon.noname-i con 0x7f48f00a16c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient(hunting): picked mon.noname-e con 0x7f48f013d050 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f00a3220 con 0x7f48f00a1d00 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f00a2c60 con 0x7f48f00a16c0 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f00a1aa0 con 0x7f48f013d050 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient(hunting): authenticate will time out at 1085.814425s 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f68e0640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f48f013d050 0x7f48f013f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54764/0 (socket says 172.21.15.3:54764) 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f60df640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f48f00a1d00 0x7f48f00a20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48880/0 (socket says 172.21.15.3:48880) 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f68e0640 1 -- 172.21.15.3:0/3838145504 learned_addr learned my addr 172.21.15.3:0/3838145504 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2582594280 0 0) 0x7f48f00a1aa0 con 0x7f48f013d050 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient(hunting): my global_id is 4688 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48dc003530 con 0x7f48f013d050 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202694875 0 0) 0x7f48f00a3220 con 0x7f48f00a1d00 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient(hunting): my global_id is 134138 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48f00a1aa0 con 0x7f48f00a1d00 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 940388421 0 0) 0x7f48f00a1aa0 con 0x7f48f00a1d00 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 >> v1:172.21.15.3:6793/0 conn(0x7f48f013d050 legacy=0x7f48f013f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 >> v1:172.21.15.203:6792/0 conn(0x7f48f00a16c0 legacy=0x7f48f013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 monclient: found mon.noname-b 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f00a1070 con 0x7f48f00a1d00 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f48ec001820 con 0x7f48f00a1d00 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 5 monclient: authenticate success, global_id 134138 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: dump: 2024-05-21T17:23:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48ec0019c0 con 0x7f48f00a1d00 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 4 set_mon_vals no callback set 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 1 -- 172.21.15.3:0/3838145504 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f48ec004b30 con 0x7f48f00a1d00 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f50dd640 10 monclient: dump: 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 10 monclient: shutdown 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- 172.21.15.3:0/3838145504 >> v1:172.21.15.3:6790/0 conn(0x7f48f00a1d00 legacy=0x7f48f00a20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- 172.21.15.3:0/3838145504 shutdown_connections 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- 172.21.15.3:0/3838145504 >> 172.21.15.3:0/3838145504 conn(0x7f48f00a72c0 msgr2=0x7f48f00a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- 172.21.15.3:0/3838145504 shutdown_connections 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.652+0000 7f48f8343640 1 -- 172.21.15.3:0/3838145504 wait complete. 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: monmap: 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.670 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 1 Processor -- start 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 1 -- start start 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: init 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: _add_conns ranks=[6,3,7,0,8,1,2,5,4] 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient(hunting): picked mon.noname-d con 0x7f48f00a16c0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient(hunting): picked mon.noname-g con 0x7f48f00a1d00 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient(hunting): picked mon.noname-i con 0x7f48f013d050 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f013ab70 con 0x7f48f00a1d00 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f0073c30 con 0x7f48f00a16c0 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f00736e0 con 0x7f48f013d050 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient(hunting): authenticate will time out at 1085.817261s 2024-05-21T17:23:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f60df640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f48f00a16c0 0x7f48f0137910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:42946/0 (socket says 172.21.15.3:42946) 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f60df640 1 -- 172.21.15.3:0/3966957128 learned_addr learned my addr 172.21.15.3:0/3966957128 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806357772 0 0) 0x7f48f0073c30 con 0x7f48f00a16c0 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): my global_id is 4497 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48d4003530 con 0x7f48f00a16c0 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 147371612 0 0) 0x7f48d4003530 con 0x7f48f00a16c0 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48f0073c30 con 0x7f48f00a16c0 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f48ec0016e0 con 0x7f48f00a16c0 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:23:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2224883654 0 0) 0x7f48f0073c30 con 0x7f48f00a16c0 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 >> v1:172.21.15.203:6792/0 conn(0x7f48f013d050 legacy=0x7f48f01d5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 >> v1:172.21.15.203:6790/0 conn(0x7f48f00a1d00 legacy=0x7f48f0138020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 monclient: found mon.d 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f0138730 con 0x7f48f00a16c0 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 5 monclient: authenticate success, global_id 4497 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: _renew_subs 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48f013b5f0 con 0x7f48f00a16c0 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: _renew_subs 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f48f01372d0 con 0x7f48f00a16c0 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48ec001980 con 0x7f48f00a16c0 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f48ec005130 con 0x7f48f00a16c0 2024-05-21T17:23:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient: dump: 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.676 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f48dc640 4 set_mon_vals no callback set 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f48ec0102b0 con 0x7f48f00a16c0 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48d80065d0 con 0x7f48d8002870 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f48ec045d40 con 0x7f48f00a16c0 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48f68e0640 10 monclient: get_auth_request con 0x7f48d4038c10 auth_method 0 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270290382 0 0) 0x7f48d80065d0 con 0x7f48d8002870 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 10 monclient(hunting): my global_id is 4712 2024-05-21T17:23:38.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.656+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48f0073c30 con 0x7f48d8002870 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1722564418 0 0) 0x7f48f0073c30 con 0x7f48d8002870 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48d80065d0 con 0x7f48d8002870 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f48e00019c0 con 0x7f48d8002870 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 10 monclient: dump: 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.681 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2923949932 0 0) 0x7f48d80065d0 con 0x7f48d8002870 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.203:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48d800a170 con 0x7f48d8002870 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f48e0001cd0 con 0x7f48d8002870 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 >> v1:172.21.15.203:6791/0 conn(0x7f48d8002870 legacy=0x7f48d8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.660+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 >> v1:172.21.15.203:6791/0 conn(0x7f48d8002870 legacy=0x7f48d8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.664+0000 7f48f8343640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.664+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48d8009e80 con 0x7f48d8009140 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778352555 0 0) 0x7f48d8009e80 con 0x7f48d8009140 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient(hunting): my global_id is 4721 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48d80065d0 con 0x7f48d8009140 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3863299652 0 0) 0x7f48d80065d0 con 0x7f48d8009140 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.203:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48d8009e80 con 0x7f48d8009140 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f48e0003b30 con 0x7f48d8009140 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient: dump: 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1433895890 0 0) 0x7f48d8009e80 con 0x7f48d8009140 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 --> v1:172.21.15.203:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48d800c420 con 0x7f48d8009140 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 <== mon.5 v1:172.21.15.203:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7705 (unknown 0 0 2527025941) 0x7f48e0004fa0 con 0x7f48d8009140 2024-05-21T17:23:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 >> v1:172.21.15.203:6791/0 conn(0x7f48d8009140 legacy=0x7f48d80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48e6ffd640 1 -- 172.21.15.3:0/3966957128 >> v1:172.21.15.203:6791/0 conn(0x7f48d8009140 legacy=0x7f48d80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:38.685 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 142567, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:23:38.686 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.687 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.688 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:38.689 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.690 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:38.691 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.694 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.695 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.696 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.697 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.700 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.701 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.704 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 2 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:38.705 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 >> v1:172.21.15.203:6812/343827626 conn(0x7f48d4038c10 legacy=0x7f48d403b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f8343640 10 monclient: shutdown 2024-05-21T17:23:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f8343640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 >> v1:172.21.15.3:6792/0 conn(0x7f48f00a16c0 legacy=0x7f48f0137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f68e0640 1 -- 172.21.15.3:0/3966957128 reap_dead start 2024-05-21T17:23:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f68e0640 1 -- 172.21.15.3:0/3966957128 reap_dead start 2024-05-21T17:23:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 shutdown_connections 2024-05-21T17:23:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 >> 172.21.15.3:0/3966957128 conn(0x7f48f00a72c0 msgr2=0x7f48f00a5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 shutdown_connections 2024-05-21T17:23:38.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.668+0000 7f48f8343640 1 -- 172.21.15.3:0/3966957128 wait complete. 2024-05-21T17:23:38.712 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:38.712 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-05-21T17:23:38.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.812+0000 7ff918223640 10 monclient: get_monmap_and_config 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.812+0000 7ff918223640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.812+0000 7ff918223640 10 monclient: monmap: 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:38.819610+0000 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:38.819610+0000 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.818 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.819 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 Processor -- start 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- start start 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: init 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: _add_conns ranks=[4,8,0,3,1,2,7,6,5] 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): picked mon.noname-c con 0x7ff9100a3b40 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): picked mon.noname-e con 0x7ff9100a4630 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): picked mon.noname-a con 0x7ff91009ea30 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff91009ee70 con 0x7ff91009ea30 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9100a50a0 con 0x7ff9100a3b40 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9100a52e0 con 0x7ff9100a4630 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): authenticate will time out at 1085.976937s 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff9157be640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7ff9100a4630 0x7ff91013bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54780/0 (socket says 172.21.15.3:54780) 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff9157be640 1 -- 172.21.15.3:0/874132003 learned_addr learned my addr 172.21.15.3:0/874132003 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff9167c0640 1 --1- 172.21.15.3:0/874132003 >> v1:172.21.15.3:6789/0 conn(0x7ff91009ea30 0x7ff91013f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55078/0 (socket says 172.21.15.3:55078) 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181407131 0 0) 0x7ff9100a52e0 con 0x7ff9100a4630 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): my global_id is 4697 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8fc003530 con 0x7ff9100a4630 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2087220084 0 0) 0x7ff91009ee70 con 0x7ff91009ea30 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): my global_id is 4671 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9100a52e0 con 0x7ff91009ea30 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 486963778 0 0) 0x7ff9100a50a0 con 0x7ff9100a3b40 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient(hunting): my global_id is 4657 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff91009ee70 con 0x7ff9100a3b40 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3061489686 0 0) 0x7ff8fc003530 con 0x7ff9100a4630 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 >> v1:172.21.15.3:6791/0 conn(0x7ff9100a3b40 legacy=0x7ff9100a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 >> v1:172.21.15.3:6789/0 conn(0x7ff91009ea30 legacy=0x7ff91013f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 monclient: found mon.noname-e 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9100a9ae0 con 0x7ff9100a4630 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff900001840 con 0x7ff9100a4630 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: dump: 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.831 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 5 monclient: authenticate success, global_id 4697 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9000019e0 con 0x7ff9100a4630 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 4 set_mon_vals no callback set 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 1 -- 172.21.15.3:0/874132003 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff900004bf0 con 0x7ff9100a4630 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff914fbd640 10 monclient: dump: 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: shutdown 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- 172.21.15.3:0/874132003 >> v1:172.21.15.3:6793/0 conn(0x7ff9100a4630 legacy=0x7ff91013bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- 172.21.15.3:0/874132003 shutdown_connections 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- 172.21.15.3:0/874132003 >> 172.21.15.3:0/874132003 conn(0x7ff910068c60 msgr2=0x7ff9100a9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- 172.21.15.3:0/874132003 shutdown_connections 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- 172.21.15.3:0/874132003 wait complete. 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: monmap: 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:38.834 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 Processor -- start 2024-05-21T17:23:38.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- start start 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: init 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient: _add_conns ranks=[2,0,4,5,1,7,3,6,8] 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): picked mon.noname-b con 0x7ff91009ea30 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): picked mon.noname-a con 0x7ff9100a3b40 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): picked mon.noname-c con 0x7ff9100a4630 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff91009f730 con 0x7ff9100a3b40 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff91009d8b0 con 0x7ff91009ea30 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff91013fc50 con 0x7ff9100a4630 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff918223640 10 monclient(hunting): authenticate will time out at 1085.979897s 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff9157be640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff9100a3b40 0x7ff91013e9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55082/0 (socket says 172.21.15.3:55082) 2024-05-21T17:23:38.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.816+0000 7ff9157be640 1 -- 172.21.15.3:0/585452851 learned_addr learned my addr 172.21.15.3:0/585452851 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973621781 0 0) 0x7ff91013fc50 con 0x7ff9100a4630 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): my global_id is 4666 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8f4003560 con 0x7ff9100a4630 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979675060 0 0) 0x7ff91009d8b0 con 0x7ff91009ea30 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): my global_id is 134147 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff91013fc50 con 0x7ff91009ea30 2024-05-21T17:23:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911634051 0 0) 0x7ff91009f730 con 0x7ff9100a3b40 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): my global_id is 4680 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff91009d8b0 con 0x7ff9100a3b40 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2785910592 0 0) 0x7ff8f4003560 con 0x7ff9100a4630 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff91009f730 con 0x7ff9100a4630 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff908001e60 con 0x7ff9100a4630 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): dump: 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 18825493 0 0) 0x7ff91013fc50 con 0x7ff91009ea30 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8f4003560 con 0x7ff91009ea30 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3828715185 0 0) 0x7ff91009d8b0 con 0x7ff9100a3b40 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff91013fc50 con 0x7ff9100a3b40 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff90c001560 con 0x7ff91009ea30 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): dump: 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff900001800 con 0x7ff9100a3b40 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): dump: 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1089919151 0 0) 0x7ff91009f730 con 0x7ff9100a4630 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 >> v1:172.21.15.3:6790/0 conn(0x7ff91009ea30 legacy=0x7ff91013e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 >> v1:172.21.15.3:6789/0 conn(0x7ff9100a3b40 legacy=0x7ff91013e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 monclient: found mon.c 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff910124cc0 con 0x7ff9100a4630 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 5 monclient: authenticate success, global_id 4666 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 10 monclient: _renew_subs 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff910124340 con 0x7ff9100a4630 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 10 monclient: _renew_subs 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ff910140980 con 0x7ff9100a4630 2024-05-21T17:23:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff908002000 con 0x7ff9100a4630 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff908005160 con 0x7ff9100a4630 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff907fff640 4 set_mon_vals no callback set 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: dump: 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff8f80065d0 con 0x7ff8f8002870 2024-05-21T17:23:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7ff908005380 con 0x7ff9100a4630 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7ff908045f20 con 0x7ff9100a4630 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9157be640 10 monclient: get_auth_request con 0x7ff8f4039710 auth_method 0 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884216685 0 0) 0x7ff8f80065d0 con 0x7ff8f8002870 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient(hunting): my global_id is 4723 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff91009f730 con 0x7ff8f8002870 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1978143120 0 0) 0x7ff91009f730 con 0x7ff8f8002870 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8f80065d0 con 0x7ff8f8002870 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff90c0017f0 con 0x7ff8f8002870 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: dump: 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2087620784 0 0) 0x7ff8f80065d0 con 0x7ff8f8002870 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.203:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8f800a170 con 0x7ff8f8002870 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff90c005bb0 con 0x7ff8f8002870 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 >> v1:172.21.15.203:6792/0 conn(0x7ff8f8002870 legacy=0x7ff8f8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.820+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 >> v1:172.21.15.203:6792/0 conn(0x7ff8f8002870 legacy=0x7ff8f8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff918223640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff8f8009e30 con 0x7ff8f80090f0 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1644910770 0 0) 0x7ff8f8009e30 con 0x7ff8f80090f0 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 10 monclient(hunting): my global_id is 4732 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8f80065d0 con 0x7ff8f80090f0 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2838507869 0 0) 0x7ff8f80065d0 con 0x7ff8f80090f0 2024-05-21T17:23:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.203:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8f8009e30 con 0x7ff8f80090f0 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7ff90c004330 con 0x7ff8f80090f0 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 10 monclient: dump: 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4293609398 0 0) 0x7ff8f8009e30 con 0x7ff8f80090f0 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.828+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 --> v1:172.21.15.203:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff8f800c400 con 0x7ff8f80090f0 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 <== mon.7 v1:172.21.15.203:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7875 (unknown 0 0 289961283) 0x7ff90c004590 con 0x7ff8f80090f0 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff9067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff9067fc640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 >> v1:172.21.15.203:6792/0 conn(0x7ff8f80090f0 legacy=0x7ff8f8009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff9067fc640 1 -- 172.21.15.3:0/585452851 >> v1:172.21.15.203:6792/0 conn(0x7ff8f80090f0 legacy=0x7ff8f8009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:38.854 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:38.854 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:38.854 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:38.854 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "peon", 2024-05-21T17:23:38.854 INFO:teuthology.orchestra.run.smithi003.stdout: "election_epoch": 20, 2024-05-21T17:23:38.854 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 142711, 2024-05-21T17:23:38.854 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum": [ 2024-05-21T17:23:38.854 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: 8 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_age": 1, 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: "required_con": "2450521708007866372", 2024-05-21T17:23:38.855 INFO:teuthology.orchestra.run.smithi003.stdout: "required_mon": [ 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_con": "4540701547738038271", 2024-05-21T17:23:38.856 INFO:teuthology.orchestra.run.smithi003.stdout: "quorum_mon": [ 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "outside_quorum": [], 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "extra_probe_peers": [], 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "sync_provider": [], 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": { 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 1, 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "4db3b12d-01c4-4d6c-b29b-1648ee37f87f", 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-05-21T17:20:49.979435Z", 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release": 19, 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "min_mon_release_name": "squid", 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "election_strategy": 3, 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "disallowed_leaders: ": "", 2024-05-21T17:23:38.858 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false, 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "tiebreaker_mon": "", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_ranks: ": "", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "features": { 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "persistent": [ 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "kraken", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "luminous", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "mimic", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap-prune", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "nautilus", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific", 2024-05-21T17:23:38.859 INFO:teuthology.orchestra.run.smithi003.stdout: "elector-pinging", 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy", 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "reef", 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "squid" 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "optional": [] 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "mons": [ 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 0, 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "a", 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789", 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6789/0", 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 1, 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "f", 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789", 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.862 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6789/0", 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 2, 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "b", 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790", 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6790/0", 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 3, 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "g", 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.864 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790", 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6790/0", 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 4, 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "c", 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791", 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.866 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6791/0", 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 5, 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "h", 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791", 2024-05-21T17:23:38.867 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6791/0", 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 6, 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "d", 2024-05-21T17:23:38.868 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792", 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6792/0", 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 7, 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "i", 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792", 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.870 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.203:6792/0", 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "rank": 8, 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "e", 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-05-21T17:23:38.871 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793", 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 0 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6793/0", 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: "priority": 0, 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0, 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "{}" 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: "feature_map": { 2024-05-21T17:23:38.872 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": [ 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": [ 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.873 INFO:teuthology.orchestra.run.smithi003.stdout: "client": [ 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": [ 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: "features": "0x3f03cffffffdffff", 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: "release": "luminous", 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 1 2024-05-21T17:23:38.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-21T17:23:38.875 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-21T17:23:38.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-21T17:23:38.875 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": false 2024-05-21T17:23:38.875 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 >> v1:172.21.15.203:6812/343827626 conn(0x7ff8f4039710 legacy=0x7ff8f403bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff918223640 10 monclient: shutdown 2024-05-21T17:23:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff918223640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 >> v1:172.21.15.3:6791/0 conn(0x7ff9100a4630 legacy=0x7ff9101d5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff9167c0640 1 -- 172.21.15.3:0/585452851 reap_dead start 2024-05-21T17:23:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff9167c0640 1 -- 172.21.15.3:0/585452851 reap_dead start 2024-05-21T17:23:38.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 shutdown_connections 2024-05-21T17:23:38.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 >> 172.21.15.3:0/585452851 conn(0x7ff910068c60 msgr2=0x7ff91013bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:38.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 shutdown_connections 2024-05-21T17:23:38.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.832+0000 7ff918223640 1 -- 172.21.15.3:0/585452851 wait complete. 2024-05-21T17:23:38.877 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-05-21T17:23:38.877 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-21T17:23:38.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-21T17:23:38.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.984+0000 7f111ceb0640 10 monclient: get_monmap_and_config 2024-05-21T17:23:38.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.984+0000 7f111ceb0640 10 monclient: build_initial_monmap 2024-05-21T17:23:38.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.984+0000 7f111ceb0640 10 monclient: monmap: 2024-05-21T17:23:38.989 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:38.989 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:38.989 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:38.990855+0000 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:38.990855+0000 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:38.990 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.984+0000 7f111ceb0640 1 Processor -- start 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.984+0000 7f111ceb0640 1 -- start start 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.984+0000 7f111ceb0640 10 monclient: init 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: _add_conns ranks=[1,5,2,6,8,3,0,7,4] 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): picked mon.noname-f con 0x7f11181356d0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): picked mon.noname-h con 0x7f111806e000 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): picked mon.noname-b con 0x7f111813f240 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1118135280 con 0x7f11181356d0 2024-05-21T17:23:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111806d090 con 0x7f111813f240 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111806d650 con 0x7f111806e000 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): authenticate will time out at 1086.148413s 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1116d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f111813f240 0x7f1118141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48898/0 (socket says 172.21.15.3:48898) 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1116d9d640 1 -- 172.21.15.3:0/3850996703 learned_addr learned my addr 172.21.15.3:0/3850996703 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405503393 0 0) 0x7f111806d090 con 0x7f111813f240 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient(hunting): my global_id is 134156 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1100003530 con 0x7f111813f240 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 976149423 0 0) 0x7f1100003530 con 0x7f111813f240 2024-05-21T17:23:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 >> v1:172.21.15.203:6791/0 conn(0x7f111806e000 legacy=0x7f111813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 >> v1:172.21.15.203:6789/0 conn(0x7f11181356d0 legacy=0x7f1118135ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: _finish_hunting 0 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 monclient: found mon.noname-b 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111806e3e0 con 0x7f111813f240 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: _finish_auth 0 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: _check_auth_tickets 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1108001840 con 0x7f111813f240 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:39.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: dump: 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:39.001 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 5 monclient: authenticate success, global_id 134156 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1108001a60 con 0x7f111813f240 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 1 -- 172.21.15.3:0/3850996703 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1108001d50 con 0x7f111813f240 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 4 set_mon_vals no callback set 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:39.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111559a640 10 monclient: dump: 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: shutdown 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- 172.21.15.3:0/3850996703 >> v1:172.21.15.3:6790/0 conn(0x7f111813f240 legacy=0x7f1118141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- 172.21.15.3:0/3850996703 shutdown_connections 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- 172.21.15.3:0/3850996703 >> 172.21.15.3:0/3850996703 conn(0x7f1118068c60 msgr2=0x7f11180a9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- 172.21.15.3:0/3850996703 shutdown_connections 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- 172.21.15.3:0/3850996703 wait complete. 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: build_initial_monmap 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: monmap: 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:39.004 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:39.005 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 Processor -- start 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- start start 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: init 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient: _add_conns ranks=[1,0,3,7,8,4,6,5,2] 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): picked mon.noname-f con 0x7f111806e000 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): picked mon.noname-a con 0x7f11181356d0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): picked mon.noname-g con 0x7f111813f240 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1118132aa0 con 0x7f11181356d0 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111809e6c0 con 0x7f111806e000 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111809ec80 con 0x7f111813f240 2024-05-21T17:23:39.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f111ceb0640 10 monclient(hunting): authenticate will time out at 1086.151479s 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1115d9b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f11181356d0 0x7f1118132390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55088/0 (socket says 172.21.15.3:55088) 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1115d9b640 1 -- 172.21.15.3:0/2069854629 learned_addr learned my addr 172.21.15.3:0/2069854629 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418516269 0 0) 0x7f1118132aa0 con 0x7f11181356d0 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1106ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1106ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1106ffd640 10 monclient(hunting): my global_id is 4689 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.988+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f8003530 con 0x7f11181356d0 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3261200121 0 0) 0x7f10f8003530 con 0x7f11181356d0 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1118132aa0 con 0x7f11181356d0 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f110c001b80 con 0x7f11181356d0 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175699047 0 0) 0x7f111809e6c0 con 0x7f111806e000 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): my global_id is 134155 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f8003530 con 0x7f111806e000 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321238142 0 0) 0x7f111809ec80 con 0x7f111813f240 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:39.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): my global_id is 4665 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f111809e6c0 con 0x7f111813f240 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2384919015 0 0) 0x7f1118132aa0 con 0x7f11181356d0 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.203:6790/0 conn(0x7f111813f240 legacy=0x7f11181d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.203:6789/0 conn(0x7f111806e000 legacy=0x7f1118131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 monclient: found mon.a 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111809e9a0 con 0x7f11181356d0 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 5 monclient: authenticate success, global_id 4689 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 10 monclient: _renew_subs 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1118132e70 con 0x7f11181356d0 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 10 monclient: _renew_subs 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f11180a2f30 con 0x7f11181356d0 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1115d9b640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.3:6789/0 conn(0x7f11181356d0 legacy=0x7f1118132390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1115d9b640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.3:6789/0 conn(0x7f11181356d0 legacy=0x7f1118132390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1115d9b640 1 --1- 172.21.15.3:0/2069854629 >> v1:172.21.15.3:6789/0 conn(0x7f11181356d0 0x7f1118132390 :-1 s=OPENED pgs=159 cs=1 l=1).handle_message read tag failed 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1115d9b640 1 --1- 172.21.15.3:0/2069854629 >> v1:172.21.15.3:6789/0 conn(0x7f11181356d0 0x7f1118132390 :-1 s=OPENED pgs=159 cs=1 l=1).fault on lossy channel, failing 2024-05-21T17:23:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.3:6789/0 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.3:6789/0 conn(0x7f11181356d0 legacy=0x7f1118132390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _add_conns ranks=[0,6,1,5,7,2,8,3,4] 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): picked mon.a con 0x7f10f8005140 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): picked mon.d con 0x7f10f80090d0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): picked mon.f con 0x7f111806e000 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1118132aa0 con 0x7f10f8005140 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f111809ec80 con 0x7f111806e000 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f10f8012d60 con 0x7f10f80090d0 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 649003600 0 0) 0x7f1118132aa0 con 0x7f10f8005140 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f10f8012f30 con 0x7f10f8005140 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989509161 0 0) 0x7f10f8012d60 con 0x7f10f80090d0 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6792/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f1118132aa0 con 0x7f10f80090d0 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 3189047574 0 0) 0x7f10f8012f30 con 0x7f10f8005140 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.3:6792/0 conn(0x7f10f80090d0 legacy=0x7f10f800b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.203:6789/0 conn(0x7f111806e000 legacy=0x7f10f800f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 monclient: found mon.a 2024-05-21T17:23:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f10f8014180 con 0x7f10f8005140 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1110001ce0 con 0x7f10f8005140 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: dump: 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:39.015 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111809efd0 con 0x7f10f8005140 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1110001e80 con 0x7f10f8005140 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1107fff640 4 set_mon_vals no callback set 2024-05-21T17:23:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f111000e960 con 0x7f10f8005140 2024-05-21T17:23:39.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 6 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f11100443b0 con 0x7f10f8005140 2024-05-21T17:23:39.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.992+0000 7f1115d9b640 10 monclient: get_auth_request con 0x7f11181356d0 auth_method 0 2024-05-21T17:23:39.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.996+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f11100144d0 con 0x7f10f8005140 2024-05-21T17:23:39.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.996+0000 7f1106ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:39.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:38.996+0000 7f1106ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:39.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.224+0000 7f111ceb0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-05-21T17:23:39.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.224+0000 7f111ceb0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-05-21T17:23:39.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.224+0000 7f111ceb0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:39.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.224+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f111806bf50 con 0x7f10f8005140 2024-05-21T17:23:39.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f1106ffd640 1 -- 172.21.15.3:0/2069854629 <== mon.0 v1:172.21.15.3:6789/0 8 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f1110013dd0 con 0x7f10f8005140 2024-05-21T17:23:39.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f1106ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-05-21T17:23:39.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f1106ffd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:39.234 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.234 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=3492468830,config=3541143443,health=3977944326,kv=1364725586,logm=2062674972}) 2024-05-21T17:23:39.234 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=10,logm=58} crc {auth=2152950331,config=3541143443,health=2008683352,kv=2621466335,logm=2061580005}) 2024-05-21T17:23:39.234 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.234 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=3492468830,config=3541143443,health=3977944326,kv=1364725586,logm=2062674972}) 2024-05-21T17:23:39.234 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=4043417832,config=3541143443,health=548316829,kv=2533687128,logm=2318019467}) 2024-05-21T17:23:39.234 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=3492468830,config=3541143443,health=3977944326,kv=1364725586,logm=2062674972}) 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=20,config=2,health=11,kv=11,logm=56} crc {auth=2626301756,config=3541143443,health=3926953126,kv=1364725586,logm=3112819313}) 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=3492468830,config=3541143443,health=3977944326,kv=1364725586,logm=2062674972}) 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=21,config=2,health=12,kv=11,logm=54} crc {auth=3618455533,config=3541143443,health=3977944326,kv=1364725586,logm=1064549180}) 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=3492468830,config=3541143443,health=3977944326,kv=1364725586,logm=2062674972}) 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=21,config=1,health=11,kv=10,logm=57} crc {auth=3618455533,config=555582061,health=548468587,kv=3549839102,logm=2028745561}) 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=3492468830,config=3541143443,health=3977944326,kv=1364725586,logm=2062674972}) 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=725507279,config=3541143443,health=3977944326,kv=1263541200,logm=3281757940}) 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.235 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=3492468830,config=3541143443,health=3977944326,kv=1364725586,logm=2062674972}) 2024-05-21T17:23:39.236 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=21,config=2,health=11,kv=9,logm=57} crc {auth=3796116455,config=3541143443,health=2495069949,kv=649316956,logm=354610652}) 2024-05-21T17:23:39.236 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.236 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=3492468830,config=3541143443,health=3977944326,kv=1364725586,logm=2062674972}) 2024-05-21T17:23:39.236 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=1216385138,config=3541143443,health=3977944326,kv=1364725586,logm=989837343}) 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.228+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.203:6812/343827626 conn(0x7f11181356d0 legacy=0x7f10f8047010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f111ceb0640 10 monclient: shutdown 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f111ceb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 >> v1:172.21.15.3:6789/0 conn(0x7f10f8005140 legacy=0x7f10f80079b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f1116d9d640 1 -- 172.21.15.3:0/2069854629 reap_dead start 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 shutdown_connections 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 >> 172.21.15.3:0/2069854629 conn(0x7f1118068c60 msgr2=0x7f11181349f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 shutdown_connections 2024-05-21T17:23:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f111ceb0640 1 -- 172.21.15.3:0/2069854629 wait complete. 2024-05-21T17:23:39.241 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.241 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765882255}) 2024-05-21T17:23:39.241 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=949746350}) 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765882255}) 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295894377}) 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765882255}) 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2884389200}) 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765882255}) 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=654549305}) 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.242 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765882255}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2932494521}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765882255}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1717920088}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765882255}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1131703394}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765882255}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.232+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3767576457}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553745210}) 2024-05-21T17:23:39.243 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1600630798}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553745210}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2985573067}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553745210}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=670440644}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553745210}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3901443960}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553745210}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3797411991}) 2024-05-21T17:23:39.244 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.245 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553745210}) 2024-05-21T17:23:39.245 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2152944196}) 2024-05-21T17:23:39.245 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.245 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553745210}) 2024-05-21T17:23:39.245 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984044115}) 2024-05-21T17:23:39.245 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.245 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553745210}) 2024-05-21T17:23:39.245 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.236+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3493744908}) 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062939853}) 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011465527}) 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062939853}) 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=404204853}) 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062939853}) 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3509642671}) 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.250 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062939853}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=546353798}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062939853}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=497553747}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062939853}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2298473351}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062939853}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3275973377}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062939853}) 2024-05-21T17:23:39.251 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.240+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=907662043}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545646}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180201595}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545646}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4072252769}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545646}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2940602008}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545646}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2250074820}) 2024-05-21T17:23:39.252 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545646}) 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2171932797}) 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545646}) 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2208607596}) 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545646}) 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1654335124}) 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545646}) 2024-05-21T17:23:39.253 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.244+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3803876177}) 2024-05-21T17:23:39.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-05-21T17:23:39.256 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042781626}) 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719772834}) 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042781626}) 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3066535330}) 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042781626}) 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1272559924}) 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042781626}) 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3519036412}) 2024-05-21T17:23:39.257 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042781626}) 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3837860223}) 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042781626}) 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1441796046}) 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042781626}) 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3389551851}) 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042781626}) 2024-05-21T17:23:39.258 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.248+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1596791287}) 2024-05-21T17:23:39.263 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.263 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3346528313,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2212435783}) 2024-05-21T17:23:39.263 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=23,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osdmap=31} crc {logm=635363112,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osdmap=757333489}) 2024-05-21T17:23:39.263 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.263 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3346528313,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2212435783}) 2024-05-21T17:23:39.263 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=33} crc {logm=847995881,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=2239811020,mon_config_key=131999861,monmap=1493019792,osd_metadata=1017583716,osd_pg_creating=1197348261,osdmap=2303075762}) 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3346528313,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2212435783}) 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=22,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=35} crc {logm=374843674,mdsmap=2840952586,mgr=2770231248,mgr_metadata=4059471874,mgrstat=1786969116,mon_config_key=647449359,monmap=1558424919,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2766533378}) 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3346528313,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2212435783}) 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=33} crc {logm=3873425711,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=1082543782}) 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3346528313,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2212435783}) 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=33} crc {logm=847995881,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=279286338,mon_config_key=363992174,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2906941426}) 2024-05-21T17:23:39.264 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3346528313,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2212435783}) 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=37} crc {logm=1762150742,mdsmap=1401145748,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=2239811020,mon_config_key=3632734075,monmap=1493019792,osd_metadata=637844530,osd_pg_creating=1197348261,osdmap=3890257177}) 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3346528313,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2212435783}) 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3339419838,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=3546985873}) 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3346528313,mdsmap=2840952586,mgr=1656224852,mgr_command_descs=711058943,mgr_metadata=4059471874,mgrstat=634475816,mon_config_key=647449359,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=2212435783}) 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.252+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=33} crc {logm=324363586,mdsmap=2840952586,mgr=728938273,mgr_metadata=4059471874,mgrstat=1152929659,mon_config_key=2542078672,monmap=1493019792,osd_metadata=2681659813,osd_pg_creating=1197348261,osdmap=1080226316}) 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1214730284}) 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=9} crc {osdmap=2767375632}) 2024-05-21T17:23:39.265 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1214730284}) 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=2375205336}) 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1214730284}) 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=8} crc {osdmap=1228818788}) 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1214730284}) 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=8} crc {osdmap=750630541}) 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1214730284}) 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=9} crc {osdmap=2375205336}) 2024-05-21T17:23:39.266 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.267 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1214730284}) 2024-05-21T17:23:39.267 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=8} crc {osdmap=750630541}) 2024-05-21T17:23:39.267 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.267 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1214730284}) 2024-05-21T17:23:39.267 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=9} crc {osdmap=2375205336}) 2024-05-21T17:23:39.267 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-05-21T17:23:39.267 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1214730284}) 2024-05-21T17:23:39.267 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-21T17:23:39.256+0000 7f9cacc68640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=9} crc {osdmap=2375205336}) 2024-05-21T17:23:40.255 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-05-21T17:23:40.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-21T17:23:40.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient: get_monmap_and_config 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient: build_initial_monmap 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient: monmap: 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:40.392185+0000 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:40.392185+0000 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:40.392 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:40.392 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:40.392 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:40.392 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:40.392 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:40.392 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 Processor -- start 2024-05-21T17:23:40.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- start start 2024-05-21T17:23:40.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient: init 2024-05-21T17:23:40.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:40.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient: _add_conns ranks=[8,1,2,5,3,0,7,4,6] 2024-05-21T17:23:40.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient(hunting): picked mon.noname-e con 0x7f62480a48c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:40.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient(hunting): picked mon.noname-f con 0x7f62480a4ca0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient(hunting): picked mon.noname-b con 0x7f624813f210 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62480a88a0 con 0x7f62480a4ca0 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62480a6550 con 0x7f624813f210 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62480a28c0 con 0x7f62480a48c0 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient(hunting): authenticate will time out at 1087.549617s 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624e847640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f624813f210 0x7f6248141600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48904/0 (socket says 172.21.15.3:48904) 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624e847640 1 -- 172.21.15.3:0/302351196 learned_addr learned my addr 172.21.15.3:0/302351196 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141346169 0 0) 0x7f62480a6550 con 0x7f624813f210 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient(hunting): _negotiate 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:40.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient(hunting): my global_id is 134165 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6234003530 con 0x7f624813f210 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997786019 0 0) 0x7f62480a28c0 con 0x7f62480a48c0 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient(hunting): _negotiate 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient(hunting): my global_id is 4706 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62480a6550 con 0x7f62480a48c0 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2413269924 0 0) 0x7f6234003530 con 0x7f624813f210 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 >> v1:172.21.15.3:6793/0 conn(0x7f62480a48c0 legacy=0x7f62480a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 >> v1:172.21.15.203:6789/0 conn(0x7f62480a4ca0 legacy=0x7f624813daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: _finish_hunting 0 2024-05-21T17:23:40.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 monclient: found mon.noname-b 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62480a2ba0 con 0x7f624813f210 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: _finish_auth 0 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: _check_auth_tickets 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6240001bc0 con 0x7f624813f210 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: dump: 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 5 monclient: authenticate success, global_id 134165 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6240001de0 con 0x7f624813f210 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 1 -- 172.21.15.3:0/302351196 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f62400020d0 con 0x7f624813f210 2024-05-21T17:23:40.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f624d044640 10 monclient: dump: 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.401 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 4 set_mon_vals no callback set 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 10 monclient: shutdown 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- 172.21.15.3:0/302351196 >> v1:172.21.15.3:6790/0 conn(0x7f624813f210 legacy=0x7f6248141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- 172.21.15.3:0/302351196 shutdown_connections 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- 172.21.15.3:0/302351196 >> 172.21.15.3:0/302351196 conn(0x7f62480ab5b0 msgr2=0x7f62480a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- 172.21.15.3:0/302351196 shutdown_connections 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.388+0000 7f62502aa640 1 -- 172.21.15.3:0/302351196 wait complete. 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: build_initial_monmap 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: monmap: 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:40.403 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:40.404 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:40.404 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:40.404 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:40.404 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:40.404 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:40.404 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:40.404 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 1 Processor -- start 2024-05-21T17:23:40.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 1 -- start start 2024-05-21T17:23:40.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: init 2024-05-21T17:23:40.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:40.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: _add_conns ranks=[0,3,5,8,1,2,4,6,7] 2024-05-21T17:23:40.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient(hunting): picked mon.noname-a con 0x7f62480a48c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:40.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient(hunting): picked mon.noname-g con 0x7f62480a4ca0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient(hunting): picked mon.noname-h con 0x7f624813f210 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f624809d850 con 0x7f62480a48c0 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f624813e440 con 0x7f62480a4ca0 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f624809f920 con 0x7f624813f210 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient(hunting): authenticate will time out at 1087.553926s 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f624e046640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f62480a48c0 0x7f624809ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55096/0 (socket says 172.21.15.3:55096) 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f624e046640 1 -- 172.21.15.3:0/2344208249 learned_addr learned my addr 172.21.15.3:0/2344208249 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769976322 0 0) 0x7f624809d850 con 0x7f62480a48c0 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:40.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): my global_id is 4698 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f622c003530 con 0x7f62480a48c0 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2364728197 0 0) 0x7f622c003530 con 0x7f62480a48c0 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f624809d850 con 0x7f62480a48c0 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6238001ce0 con 0x7f62480a48c0 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): dump: 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.409 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2329218181 0 0) 0x7f624813e440 con 0x7f62480a4ca0 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient(hunting): my global_id is 4674 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f622c003530 con 0x7f62480a4ca0 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3928599439 0 0) 0x7f624809d850 con 0x7f62480a48c0 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 >> v1:172.21.15.203:6791/0 conn(0x7f624813f210 legacy=0x7f62481d5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 >> v1:172.21.15.203:6790/0 conn(0x7f62480a4ca0 legacy=0x7f624809d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:40.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 monclient: found mon.a 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624809f080 con 0x7f62480a48c0 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient: _finish_auth 0 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 5 monclient: authenticate success, global_id 4698 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: _renew_subs 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f624809e190 con 0x7f62480a48c0 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: _renew_subs 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f62480a1990 con 0x7f62480a48c0 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62380038b0 con 0x7f62480a48c0 2024-05-21T17:23:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6238003de0 con 0x7f62480a48c0 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 10 monclient: dump: 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.413 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f62502aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f624c843640 4 set_mon_vals no callback set 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.392+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f6238004040 con 0x7f62480a48c0 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.396+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f6238045c00 con 0x7f62480a48c0 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.396+0000 7f62502aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.396+0000 7f62502aa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.396+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f624806bf50 con 0x7f62480a48c0 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.396+0000 7f624d845640 10 monclient: get_auth_request con 0x7f622c038c90 auth_method 0 2024-05-21T17:23:40.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.400+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f6238015e70 con 0x7f62480a48c0 2024-05-21T17:23:40.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.400+0000 7f623effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:40.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.400+0000 7f623effd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:40.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.624+0000 7f62502aa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-05-21T17:23:40.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.624+0000 7f62502aa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-05-21T17:23:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.624+0000 7f62502aa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.624+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f62480a1f50 con 0x7f62480a48c0 2024-05-21T17:23:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.624+0000 7f623effd640 1 -- 172.21.15.3:0/2344208249 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2714 (unknown 254746657 0 2802064853) 0x7f6238015770 con 0x7f62480a48c0 2024-05-21T17:23:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.624+0000 7f623effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-05-21T17:23:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.624+0000 7f623effd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:40.631 INFO:teuthology.orchestra.run.smithi003.stdout:{"election_epoch":20,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","modified":"2024-05-21T17:20:49.979435Z","created":"2024-05-21T17:20:49.979435Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6790","nonce":0}]},"addr":"172.21.15.3:6790/0","public_addr":"172.21.15.3:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6790","nonce":0}]},"addr":"172.21.15.203:6790/0","public_addr":"172.21.15.203:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6791","nonce":0}]},"addr":"172.21.15.3:6791/0","public_addr":"172.21.15.3:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6791","nonce":0}]},"addr":"172.21.15.203:6791/0","public_addr":"172.21.15.203:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6792","nonce":0}]},"addr":"172.21.15.3:6792/0","public_addr":"172.21.15.3:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6792","nonce":0}]},"addr":"172.21.15.203:6792/0","public_addr":"172.21.15.203:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6793","nonce":0}]},"addr":"172.21.15.3:6793/0","public_addr":"172.21.15.3:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-21T17:23:40.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.628+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 >> v1:172.21.15.203:6812/343827626 conn(0x7f622c038c90 legacy=0x7f622c03b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.628+0000 7f62502aa640 10 monclient: shutdown 2024-05-21T17:23:40.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.628+0000 7f62502aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.628+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 >> v1:172.21.15.3:6789/0 conn(0x7f62480a48c0 legacy=0x7f624809ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.628+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 shutdown_connections 2024-05-21T17:23:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.628+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 >> 172.21.15.3:0/2344208249 conn(0x7f62480ab5b0 msgr2=0x7f6248141600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.628+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 shutdown_connections 2024-05-21T17:23:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.628+0000 7f62502aa640 1 -- 172.21.15.3:0/2344208249 wait complete. 2024-05-21T17:23:40.649 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-05-21T17:23:40.649 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-21T17:23:40.659 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-21T17:23:40.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:23:40.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.772+0000 7fca2221b640 10 monclient: get_monmap_and_config 2024-05-21T17:23:40.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.772+0000 7fca2221b640 10 monclient: build_initial_monmap 2024-05-21T17:23:40.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.772+0000 7fca2221b640 10 monclient: monmap: 2024-05-21T17:23:40.778 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:40.778 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.778 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:40.778841+0000 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:40.778841+0000 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:40.779 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:40.780 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.772+0000 7fca2221b640 1 Processor -- start 2024-05-21T17:23:40.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.772+0000 7fca2221b640 1 -- start start 2024-05-21T17:23:40.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.772+0000 7fca2221b640 10 monclient: init 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: _add_conns ranks=[4,3,1,5,7,6,2,0,8] 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): picked mon.noname-c con 0x7fca1c0a6090 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): picked mon.noname-g con 0x7fca1c0a30a0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): picked mon.noname-f con 0x7fca1c0a0f60 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1c0a0590 con 0x7fca1c0a0f60 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1c0a1530 con 0x7fca1c0a30a0 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1c06cd60 con 0x7fca1c0a6090 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): authenticate will time out at 1087.936270s 2024-05-21T17:23:40.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1b7fe640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fca1c0a6090 0x7fca1c0a2990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:56914/0 (socket says 172.21.15.3:56914) 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1b7fe640 1 -- 172.21.15.3:0/4244717384 learned_addr learned my addr 172.21.15.3:0/4244717384 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 801746769 0 0) 0x7fca1c06cd60 con 0x7fca1c0a6090 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient(hunting): my global_id is 4675 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca04003530 con 0x7fca1c0a6090 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3447479496 0 0) 0x7fca04003530 con 0x7fca1c0a6090 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 >> v1:172.21.15.203:6790/0 conn(0x7fca1c0a30a0 legacy=0x7fca1c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 >> v1:172.21.15.203:6789/0 conn(0x7fca1c0a0f60 legacy=0x7fca1c1413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:40.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 monclient: found mon.noname-c 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1c06dfa0 con 0x7fca1c0a6090 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca0c001bc0 con 0x7fca1c0a6090 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: dump: 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.789 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 5 monclient: authenticate success, global_id 4675 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca0c001de0 con 0x7fca1c0a6090 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 1 -- 172.21.15.3:0/4244717384 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca0c004f70 con 0x7fca1c0a6090 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1a7fc640 10 monclient: dump: 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.791 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.792 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 4 set_mon_vals no callback set 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: shutdown 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- 172.21.15.3:0/4244717384 >> v1:172.21.15.3:6791/0 conn(0x7fca1c0a6090 legacy=0x7fca1c0a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- 172.21.15.3:0/4244717384 shutdown_connections 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- 172.21.15.3:0/4244717384 >> 172.21.15.3:0/4244717384 conn(0x7fca1c0a9480 msgr2=0x7fca1c0a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- 172.21.15.3:0/4244717384 shutdown_connections 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- 172.21.15.3:0/4244717384 wait complete. 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: build_initial_monmap 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: monmap: 2024-05-21T17:23:40.793 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 Processor -- start 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- start start 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: init 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient: _add_conns ranks=[8,0,2,3,5,4,6,1,7] 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): picked mon.noname-e con 0x7fca1c0a0f60 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): picked mon.noname-a con 0x7fca1c0a30a0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): picked mon.noname-b con 0x7fca1c0a6090 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1c141f10 con 0x7fca1c0a30a0 2024-05-21T17:23:40.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1c06d290 con 0x7fca1c0a6090 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1c073fc0 con 0x7fca1c0a0f60 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca2221b640 10 monclient(hunting): authenticate will time out at 1087.939313s 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1bfff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fca1c0a6090 0x7fca1c1d5c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48908/0 (socket says 172.21.15.3:48908) 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca1bfff640 1 -- 172.21.15.3:0/4028428102 learned_addr learned my addr 172.21.15.3:0/4028428102 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 126770886 0 0) 0x7fca1c06d290 con 0x7fca1c0a6090 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 10 monclient(hunting): my global_id is 134174 2024-05-21T17:23:40.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca08003530 con 0x7fca1c0a6090 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2498792217 0 0) 0x7fca1c141f10 con 0x7fca1c0a30a0 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 10 monclient(hunting): my global_id is 4707 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.776+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca1c06d290 con 0x7fca1c0a30a0 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3464813711 0 0) 0x7fca1c073fc0 con 0x7fca1c0a0f60 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): my global_id is 4715 2024-05-21T17:23:40.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca1c141f10 con 0x7fca1c0a0f60 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 899767904 0 0) 0x7fca08003530 con 0x7fca1c0a6090 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca1c073fc0 con 0x7fca1c0a6090 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca14001ac0 con 0x7fca1c0a6090 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): dump: 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.798 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.800 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:40.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 827220597 0 0) 0x7fca1c06d290 con 0x7fca1c0a30a0 2024-05-21T17:23:40.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca08003530 con 0x7fca1c0a30a0 2024-05-21T17:23:40.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca100019d0 con 0x7fca1c0a30a0 2024-05-21T17:23:40.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:40.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:40.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient(hunting): dump: 2024-05-21T17:23:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.802 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.802 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.802 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.802 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.802 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.802 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.802 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.803 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 282733778 0 0) 0x7fca1c073fc0 con 0x7fca1c0a6090 2024-05-21T17:23:40.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 >> v1:172.21.15.3:6793/0 conn(0x7fca1c0a0f60 legacy=0x7fca1c13dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 >> v1:172.21.15.3:6789/0 conn(0x7fca1c0a30a0 legacy=0x7fca1c13e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:40.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 monclient: found mon.b 2024-05-21T17:23:40.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:40.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:40.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1c13ee10 con 0x7fca1c0a6090 2024-05-21T17:23:40.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient: _finish_auth 0 2024-05-21T17:23:40.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:40.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:40.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 5 monclient: authenticate success, global_id 134174 2024-05-21T17:23:40.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 10 monclient: _renew_subs 2024-05-21T17:23:40.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:40.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca1c141c30 con 0x7fca1c0a6090 2024-05-21T17:23:40.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 10 monclient: _renew_subs 2024-05-21T17:23:40.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:40.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fca1c13b8e0 con 0x7fca1c0a6090 2024-05-21T17:23:40.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca140038d0 con 0x7fca1c0a6090 2024-05-21T17:23:40.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:40.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca14003e80 con 0x7fca1c0a6090 2024-05-21T17:23:40.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:40.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca19ffb640 4 set_mon_vals no callback set 2024-05-21T17:23:40.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:40.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 10 monclient: dump: 2024-05-21T17:23:40.806 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:40.806 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:40.806 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.806 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:40.806 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:40.806 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:40.807 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:40.808 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:40.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:40.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fca140040e0 con 0x7fca1c0a6090 2024-05-21T17:23:40.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fca14045b90 con 0x7fca1c0a6090 2024-05-21T17:23:40.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:40.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:40.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9fc005090 con 0x7fca1c0a6090 2024-05-21T17:23:40.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.780+0000 7fca1b7fe640 10 monclient: get_auth_request con 0x7fca080392a0 auth_method 0 2024-05-21T17:23:40.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.784+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7fca14015d50 con 0x7fca1c0a6090 2024-05-21T17:23:40.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.784+0000 7fca03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:40.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.784+0000 7fca03fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:40.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.944+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc9fc000ba0 con 0x7fca080392a0 2024-05-21T17:23:40.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.948+0000 7fca03fff640 1 -- 172.21.15.3:0/4028428102 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33795 (unknown 2965378022 0 3559376941) 0x7fc9fc000ba0 con 0x7fca080392a0 2024-05-21T17:23:40.955 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:23:40.955 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:23:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.952+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 >> v1:172.21.15.203:6812/343827626 conn(0x7fca080392a0 legacy=0x7fca0803b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.952+0000 7fca2221b640 10 monclient: shutdown 2024-05-21T17:23:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.952+0000 7fca2221b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.952+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 >> v1:172.21.15.3:6790/0 conn(0x7fca1c0a6090 legacy=0x7fca1c1d5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.952+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 shutdown_connections 2024-05-21T17:23:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.952+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 >> 172.21.15.3:0/4028428102 conn(0x7fca1c0a9480 msgr2=0x7fca1c072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.952+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 shutdown_connections 2024-05-21T17:23:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:40.952+0000 7fca2221b640 1 -- 172.21.15.3:0/4028428102 wait complete. 2024-05-21T17:23:40.972 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-05-21T17:23:39.093441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163352,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067688,"statfs":{"total":579820584960,"available":579653312512,"internally_reserved":0,"allocated":1236992,"data_stored":1439006,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003165"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.377367+0000","last_change":"2024-05-21T17:21:38.226793+0000","last_active":"2024-05-21T17:22:04.377367+0000","last_peered":"2024-05-21T17:22:04.377367+0000","last_clean":"2024-05-21T17:22:04.377367+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:22:04.377367+0000","last_undegraded":"2024-05-21T17:22:04.377367+0000","last_fullsized":"2024-05-21T17:22:04.377367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T21:23:30.360529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.371254+0000","last_change":"2024-05-21T17:21:38.830568+0000","last_active":"2024-05-21T17:22:04.371254+0000","last_peered":"2024-05-21T17:22:04.371254+0000","last_clean":"2024-05-21T17:22:04.371254+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:22:04.371254+0000","last_undegraded":"2024-05-21T17:22:04.371254+0000","last_fullsized":"2024-05-21T17:22:04.371254+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T02:04:57.235930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.731426+0000","last_change":"2024-05-21T17:21:38.216293+0000","last_active":"2024-05-21T17:22:04.731426+0000","last_peered":"2024-05-21T17:22:04.731426+0000","last_clean":"2024-05-21T17:22:04.731426+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:22:04.731426+0000","last_undegraded":"2024-05-21T17:22:04.731426+0000","last_fullsized":"2024-05-21T17:22:04.731426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:53:31.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.374521+0000","last_change":"2024-05-21T17:21:38.219369+0000","last_active":"2024-05-21T17:22:04.374521+0000","last_peered":"2024-05-21T17:22:04.374521+0000","last_clean":"2024-05-21T17:22:04.374521+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:22:04.374521+0000","last_undegraded":"2024-05-21T17:22:04.374521+0000","last_fullsized":"2024-05-21T17:22:04.374521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:24:49.428951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'72","reported_seq":114,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373702+0000","last_change":"2024-05-21T17:21:38.221127+0000","last_active":"2024-05-21T17:22:04.373702+0000","last_peered":"2024-05-21T17:22:04.373702+0000","last_clean":"2024-05-21T17:22:04.373702+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:22:04.373702+0000","last_undegraded":"2024-05-21T17:22:04.373702+0000","last_fullsized":"2024-05-21T17:22:04.373702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:12:45.986667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.370522+0000","last_change":"2024-05-21T17:21:38.219803+0000","last_active":"2024-05-21T17:22:04.370522+0000","last_peered":"2024-05-21T17:22:04.370522+0000","last_clean":"2024-05-21T17:22:04.370522+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:22:04.370522+0000","last_undegraded":"2024-05-21T17:22:04.370522+0000","last_fullsized":"2024-05-21T17:22:04.370522+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:52:49.624765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.729732+0000","last_change":"2024-05-21T17:21:38.829468+0000","last_active":"2024-05-21T17:22:04.729732+0000","last_peered":"2024-05-21T17:22:04.729732+0000","last_clean":"2024-05-21T17:22:04.729732+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:22:04.729732+0000","last_undegraded":"2024-05-21T17:22:04.729732+0000","last_fullsized":"2024-05-21T17:22:04.729732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:07:14.487479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373207+0000","last_change":"2024-05-21T17:21:38.830884+0000","last_active":"2024-05-21T17:22:04.373207+0000","last_peered":"2024-05-21T17:22:04.373207+0000","last_clean":"2024-05-21T17:22:04.373207+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:22:04.373207+0000","last_undegraded":"2024-05-21T17:22:04.373207+0000","last_fullsized":"2024-05-21T17:22:04.373207+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:36:41.643111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373983+0000","last_change":"2024-05-21T17:21:38.829272+0000","last_active":"2024-05-21T17:22:04.373983+0000","last_peered":"2024-05-21T17:22:04.373983+0000","last_clean":"2024-05-21T17:22:04.373983+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:22:04.373983+0000","last_undegraded":"2024-05-21T17:22:04.373983+0000","last_fullsized":"2024-05-21T17:22:04.373983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:09:03.800931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640282,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640283,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640284,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:23:40.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:23:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.080+0000 7f1a785e6640 10 monclient: get_monmap_and_config 2024-05-21T17:23:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.080+0000 7f1a785e6640 10 monclient: build_initial_monmap 2024-05-21T17:23:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.080+0000 7f1a785e6640 10 monclient: monmap: 2024-05-21T17:23:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:41.085 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:41.086875+0000 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:41.086875+0000 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:41.086 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:41.087 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:41.087 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.080+0000 7f1a785e6640 1 Processor -- start 2024-05-21T17:23:41.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.080+0000 7f1a785e6640 1 -- start start 2024-05-21T17:23:41.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.080+0000 7f1a785e6640 10 monclient: init 2024-05-21T17:23:41.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:41.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: _add_conns ranks=[4,0,2,3,7,6,5,8,1] 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): picked mon.noname-c con 0x7f1a700a4730 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): picked mon.noname-a con 0x7f1a700a5d40 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7013f2f0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a700a6f50 con 0x7f1a700a5d40 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a700a2960 con 0x7f1a7013f2f0 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a700a2c40 con 0x7f1a700a4730 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): authenticate will time out at 1088.244429s 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a76b83640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f1a7013f2f0 0x7f1a701416e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48920/0 (socket says 172.21.15.3:48920) 2024-05-21T17:23:41.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a76382640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f1a700a4730 0x7f1a700a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:56918/0 (socket says 172.21.15.3:56918) 2024-05-21T17:23:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a76b83640 1 -- 172.21.15.3:0/935132532 learned_addr learned my addr 172.21.15.3:0/935132532 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142653034 0 0) 0x7f1a700a2960 con 0x7f1a7013f2f0 2024-05-21T17:23:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): my global_id is 134183 2024-05-21T17:23:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a5c003530 con 0x7f1a7013f2f0 2024-05-21T17:23:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225001681 0 0) 0x7f1a700a2c40 con 0x7f1a700a4730 2024-05-21T17:23:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): my global_id is 4684 2024-05-21T17:23:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a700a2960 con 0x7f1a700a4730 2024-05-21T17:23:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901911856 0 0) 0x7f1a700a6f50 con 0x7f1a700a5d40 2024-05-21T17:23:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient(hunting): my global_id is 4716 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a700a2c40 con 0x7f1a700a5d40 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 600537041 0 0) 0x7f1a5c003530 con 0x7f1a7013f2f0 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 >> v1:172.21.15.3:6791/0 conn(0x7f1a700a4730 legacy=0x7f1a700a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 >> v1:172.21.15.3:6789/0 conn(0x7f1a700a5d40 legacy=0x7f1a7013dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: _finish_hunting 0 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 monclient: found mon.noname-b 2024-05-21T17:23:41.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a700a5220 con 0x7f1a7013f2f0 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: _finish_auth 0 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: _check_auth_tickets 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1a68001bc0 con 0x7f1a7013f2f0 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: dump: 2024-05-21T17:23:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 5 monclient: authenticate success, global_id 134183 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a68001d60 con 0x7f1a7013f2f0 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 1 -- 172.21.15.3:0/935132532 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1a68004f70 con 0x7f1a7013f2f0 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:41.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a75380640 10 monclient: dump: 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.102 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 4 set_mon_vals no callback set 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: shutdown 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- 172.21.15.3:0/935132532 >> v1:172.21.15.3:6790/0 conn(0x7f1a7013f2f0 legacy=0x7f1a701416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- 172.21.15.3:0/935132532 shutdown_connections 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- 172.21.15.3:0/935132532 >> 172.21.15.3:0/935132532 conn(0x7f1a700ab610 msgr2=0x7f1a700a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- 172.21.15.3:0/935132532 shutdown_connections 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- 172.21.15.3:0/935132532 wait complete. 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: build_initial_monmap 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: monmap: 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:41.105 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 Processor -- start 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- start start 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: init 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient: _add_conns ranks=[2,7,1,5,4,8,6,3,0] 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): picked mon.noname-b con 0x7f1a700a4730 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): picked mon.noname-i con 0x7f1a700a5d40 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:41.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): picked mon.noname-f con 0x7f1a7013f2f0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a700a1d40 con 0x7f1a7013f2f0 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a70130bf0 con 0x7f1a700a4730 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a70131930 con 0x7f1a700a5d40 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a785e6640 10 monclient(hunting): authenticate will time out at 1088.247441s 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a76382640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f1a700a4730 0x7f1a7012c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48926/0 (socket says 172.21.15.3:48926) 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a76382640 1 -- 172.21.15.3:0/1199246596 learned_addr learned my addr 172.21.15.3:0/1199246596 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519469339 0 0) 0x7f1a70130bf0 con 0x7f1a700a4730 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a66ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a66ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a66ffd640 10 monclient(hunting): my global_id is 134192 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.084+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a54003530 con 0x7f1a700a4730 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1904166568 0 0) 0x7f1a54003530 con 0x7f1a700a4730 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a70130bf0 con 0x7f1a700a4730 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1a6c001c00 con 0x7f1a700a4730 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.108 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3562822144 0 0) 0x7f1a700a1d40 con 0x7f1a7013f2f0 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient(hunting): my global_id is 134182 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a54003530 con 0x7f1a7013f2f0 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3351367744 0 0) 0x7f1a70130bf0 con 0x7f1a700a4730 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 >> v1:172.21.15.203:6792/0 conn(0x7f1a700a5d40 legacy=0x7f1a701d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 >> v1:172.21.15.203:6789/0 conn(0x7f1a7013f2f0 legacy=0x7f1a701d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 monclient: found mon.b 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a70131c10 con 0x7f1a700a4730 2024-05-21T17:23:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 5 monclient: authenticate success, global_id 134192 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 10 monclient: _renew_subs 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a700a1710 con 0x7f1a700a4730 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 10 monclient: _renew_subs 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a700a0250 con 0x7f1a700a4730 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a6c004aa0 con 0x7f1a700a4730 2024-05-21T17:23:41.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f1a6c004f90 con 0x7f1a700a4730 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a74b7f640 4 set_mon_vals no callback set 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 10 monclient: dump: 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a58005090 con 0x7f1a700a4730 2024-05-21T17:23:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f1a6c010110 con 0x7f1a700a4730 2024-05-21T17:23:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f1a6c045c30 con 0x7f1a700a4730 2024-05-21T17:23:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.088+0000 7f1a75b81640 10 monclient: get_auth_request con 0x7f1a54038c60 auth_method 0 2024-05-21T17:23:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.092+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f1a6c015d50 con 0x7f1a700a4730 2024-05-21T17:23:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.092+0000 7f1a66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.092+0000 7f1a66ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:41.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.252+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a58000ba0 con 0x7f1a54038c60 2024-05-21T17:23:41.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.256+0000 7f1a66ffd640 1 -- 172.21.15.3:0/1199246596 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33795 (unknown 2965378022 0 4195560622) 0x7f1a58000ba0 con 0x7f1a54038c60 2024-05-21T17:23:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:23:41.261 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:23:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.260+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 >> v1:172.21.15.203:6812/343827626 conn(0x7f1a54038c60 legacy=0x7f1a5403b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.260+0000 7f1a785e6640 10 monclient: shutdown 2024-05-21T17:23:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.260+0000 7f1a785e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.260+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 >> v1:172.21.15.3:6790/0 conn(0x7f1a700a4730 legacy=0x7f1a7012c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.260+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 shutdown_connections 2024-05-21T17:23:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.260+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 >> 172.21.15.3:0/1199246596 conn(0x7f1a700ab610 msgr2=0x7f1a7012f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.260+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 shutdown_connections 2024-05-21T17:23:41.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.260+0000 7f1a785e6640 1 -- 172.21.15.3:0/1199246596 wait complete. 2024-05-21T17:23:41.283 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-05-21T17:23:41.093958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163352,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067688,"statfs":{"total":579820584960,"available":579653312512,"internally_reserved":0,"allocated":1236992,"data_stored":1439006,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003119"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.377367+0000","last_change":"2024-05-21T17:21:38.226793+0000","last_active":"2024-05-21T17:22:04.377367+0000","last_peered":"2024-05-21T17:22:04.377367+0000","last_clean":"2024-05-21T17:22:04.377367+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:22:04.377367+0000","last_undegraded":"2024-05-21T17:22:04.377367+0000","last_fullsized":"2024-05-21T17:22:04.377367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T21:23:30.360529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.371254+0000","last_change":"2024-05-21T17:21:38.830568+0000","last_active":"2024-05-21T17:22:04.371254+0000","last_peered":"2024-05-21T17:22:04.371254+0000","last_clean":"2024-05-21T17:22:04.371254+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:22:04.371254+0000","last_undegraded":"2024-05-21T17:22:04.371254+0000","last_fullsized":"2024-05-21T17:22:04.371254+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T02:04:57.235930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.731426+0000","last_change":"2024-05-21T17:21:38.216293+0000","last_active":"2024-05-21T17:22:04.731426+0000","last_peered":"2024-05-21T17:22:04.731426+0000","last_clean":"2024-05-21T17:22:04.731426+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:22:04.731426+0000","last_undegraded":"2024-05-21T17:22:04.731426+0000","last_fullsized":"2024-05-21T17:22:04.731426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:53:31.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.374521+0000","last_change":"2024-05-21T17:21:38.219369+0000","last_active":"2024-05-21T17:22:04.374521+0000","last_peered":"2024-05-21T17:22:04.374521+0000","last_clean":"2024-05-21T17:22:04.374521+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:22:04.374521+0000","last_undegraded":"2024-05-21T17:22:04.374521+0000","last_fullsized":"2024-05-21T17:22:04.374521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:24:49.428951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'72","reported_seq":114,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373702+0000","last_change":"2024-05-21T17:21:38.221127+0000","last_active":"2024-05-21T17:22:04.373702+0000","last_peered":"2024-05-21T17:22:04.373702+0000","last_clean":"2024-05-21T17:22:04.373702+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:22:04.373702+0000","last_undegraded":"2024-05-21T17:22:04.373702+0000","last_fullsized":"2024-05-21T17:22:04.373702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:12:45.986667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.370522+0000","last_change":"2024-05-21T17:21:38.219803+0000","last_active":"2024-05-21T17:22:04.370522+0000","last_peered":"2024-05-21T17:22:04.370522+0000","last_clean":"2024-05-21T17:22:04.370522+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:22:04.370522+0000","last_undegraded":"2024-05-21T17:22:04.370522+0000","last_fullsized":"2024-05-21T17:22:04.370522+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:52:49.624765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.729732+0000","last_change":"2024-05-21T17:21:38.829468+0000","last_active":"2024-05-21T17:22:04.729732+0000","last_peered":"2024-05-21T17:22:04.729732+0000","last_clean":"2024-05-21T17:22:04.729732+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:22:04.729732+0000","last_undegraded":"2024-05-21T17:22:04.729732+0000","last_fullsized":"2024-05-21T17:22:04.729732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:07:14.487479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373207+0000","last_change":"2024-05-21T17:21:38.830884+0000","last_active":"2024-05-21T17:22:04.373207+0000","last_peered":"2024-05-21T17:22:04.373207+0000","last_clean":"2024-05-21T17:22:04.373207+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:22:04.373207+0000","last_undegraded":"2024-05-21T17:22:04.373207+0000","last_fullsized":"2024-05-21T17:22:04.373207+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:36:41.643111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373983+0000","last_change":"2024-05-21T17:21:38.829272+0000","last_active":"2024-05-21T17:22:04.373983+0000","last_peered":"2024-05-21T17:22:04.373983+0000","last_clean":"2024-05-21T17:22:04.373983+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:22:04.373983+0000","last_undegraded":"2024-05-21T17:22:04.373983+0000","last_fullsized":"2024-05-21T17:22:04.373983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:09:03.800931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640282,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640283,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640284,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:23:41.284 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-21T17:23:41.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:23:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.392+0000 7fbaec90f640 10 monclient: get_monmap_and_config 2024-05-21T17:23:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.392+0000 7fbaec90f640 10 monclient: build_initial_monmap 2024-05-21T17:23:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.392+0000 7fbaec90f640 10 monclient: monmap: 2024-05-21T17:23:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:41.398 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.399 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:41.399498+0000 2024-05-21T17:23:41.399 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:41.399498+0000 2024-05-21T17:23:41.399 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:41.399 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.399 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:41.399 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:41.400 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:41.400 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 Processor -- start 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- start start 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient: init 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient: _add_conns ranks=[6,4,2,1,3,0,8,7,5] 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient(hunting): picked mon.noname-d con 0x7fbae80a4910 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient(hunting): picked mon.noname-c con 0x7fbae80a4cf0 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient(hunting): picked mon.noname-b con 0x7fbae80a5150 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:41.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae80a66a0 con 0x7fbae80a5150 2024-05-21T17:23:41.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae80a2930 con 0x7fbae80a4cf0 2024-05-21T17:23:41.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae80a2c10 con 0x7fbae80a4910 2024-05-21T17:23:41.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:41.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient(hunting): authenticate will time out at 1088.557243s 2024-05-21T17:23:41.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae5d9b640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fbae80a4cf0 0x7fbae813dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:56922/0 (socket says 172.21.15.3:56922) 2024-05-21T17:23:41.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae6d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fbae80a5150 0x7fbae8141300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48928/0 (socket says 172.21.15.3:48928) 2024-05-21T17:23:41.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae5d9b640 1 -- 172.21.15.3:0/3820303032 learned_addr learned my addr 172.21.15.3:0/3820303032 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:41.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae659c640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7fbae80a4910 0x7fbae80a5f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:42964/0 (socket says 172.21.15.3:42964) 2024-05-21T17:23:41.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 118708381 0 0) 0x7fbae80a2930 con 0x7fbae80a4cf0 2024-05-21T17:23:41.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): my global_id is 4693 2024-05-21T17:23:41.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbad0003530 con 0x7fbae80a4cf0 2024-05-21T17:23:41.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315746144 0 0) 0x7fbae80a2c10 con 0x7fbae80a4910 2024-05-21T17:23:41.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): my global_id is 4506 2024-05-21T17:23:41.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae80a2930 con 0x7fbae80a4910 2024-05-21T17:23:41.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1199977180 0 0) 0x7fbae80a66a0 con 0x7fbae80a5150 2024-05-21T17:23:41.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient(hunting): my global_id is 134201 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae80a2c10 con 0x7fbae80a5150 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3185640545 0 0) 0x7fbad0003530 con 0x7fbae80a4cf0 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 >> v1:172.21.15.3:6792/0 conn(0x7fbae80a4910 legacy=0x7fbae80a5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 >> v1:172.21.15.3:6790/0 conn(0x7fbae80a5150 legacy=0x7fbae8141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: _finish_hunting 0 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 monclient: found mon.noname-c 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae80a2ef0 con 0x7fbae80a4cf0 2024-05-21T17:23:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: _finish_auth 0 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: _check_auth_tickets 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbad8001840 con 0x7fbae80a4cf0 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: dump: 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.412 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 5 monclient: authenticate success, global_id 4693 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbad8001a60 con 0x7fbae80a4cf0 2024-05-21T17:23:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 1 -- 172.21.15.3:0/3820303032 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbad8004bf0 con 0x7fbae80a4cf0 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 4 set_mon_vals no callback set 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbae559a640 10 monclient: dump: 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.415 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.415 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.415 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.415 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.416 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.416 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.416 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.416 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.416 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient: shutdown 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- 172.21.15.3:0/3820303032 >> v1:172.21.15.3:6791/0 conn(0x7fbae80a4cf0 legacy=0x7fbae813dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- 172.21.15.3:0/3820303032 shutdown_connections 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- 172.21.15.3:0/3820303032 >> 172.21.15.3:0/3820303032 conn(0x7fbae80ab660 msgr2=0x7fbae80a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- 172.21.15.3:0/3820303032 shutdown_connections 2024-05-21T17:23:41.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- 172.21.15.3:0/3820303032 wait complete. 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient: build_initial_monmap 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient: monmap: 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:41.418 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 Processor -- start 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 1 -- start start 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.396+0000 7fbaec90f640 10 monclient: init 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient: _add_conns ranks=[2,6,1,5,8,7,0,3,4] 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient(hunting): picked mon.noname-b con 0x7fbae80a4910 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient(hunting): picked mon.noname-d con 0x7fbae80a4cf0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient(hunting): picked mon.noname-f con 0x7fbae80a5150 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae80a71a0 con 0x7fbae80a5150 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae813eaf0 con 0x7fbae80a4910 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae813d760 con 0x7fbae80a4cf0 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient(hunting): authenticate will time out at 1088.560411s 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbae659c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fbae80a4910 0x7fbae809e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48938/0 (socket says 172.21.15.3:48938) 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbae659c640 1 -- 172.21.15.3:0/665201256 learned_addr learned my addr 172.21.15.3:0/665201256 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2406127609 0 0) 0x7fbae813eaf0 con 0x7fbae80a4910 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): my global_id is 134210 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbac8003530 con 0x7fbae80a4910 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1795533245 0 0) 0x7fbae813d760 con 0x7fbae80a4cf0 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): my global_id is 4515 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae813eaf0 con 0x7fbae80a4cf0 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1311207078 0 0) 0x7fbac8003530 con 0x7fbae80a4910 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbae813d760 con 0x7fbae80a4910 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbadc001b30 con 0x7fbae80a4910 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1327085346 0 0) 0x7fbae813eaf0 con 0x7fbae80a4cf0 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbac8003530 con 0x7fbae80a4cf0 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbad8004b30 con 0x7fbae80a4cf0 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477539895 0 0) 0x7fbae80a2c10 con 0x7fbae80a5150 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient(hunting): my global_id is 134191 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae813eaf0 con 0x7fbae80a5150 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 168783086 0 0) 0x7fbae813d760 con 0x7fbae80a4910 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 >> v1:172.21.15.3:6792/0 conn(0x7fbae80a4cf0 legacy=0x7fbae809ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 >> v1:172.21.15.203:6789/0 conn(0x7fbae80a5150 legacy=0x7fbae81d5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 monclient: found mon.b 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae80a1d40 con 0x7fbae80a4910 2024-05-21T17:23:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 5 monclient: authenticate success, global_id 134210 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient: _renew_subs 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbae813e810 con 0x7fbae80a4910 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient: _renew_subs 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbae813e190 con 0x7fbae80a4910 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbadc004aa0 con 0x7fbae80a4910 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fbadc005050 con 0x7fbae80a4910 2024-05-21T17:23:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbae4d99640 4 set_mon_vals no callback set 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 10 monclient: dump: 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbacc005090 con 0x7fbae80a4910 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fbadc0051f0 con 0x7fbae80a4910 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fbadc045c30 con 0x7fbae80a4910 2024-05-21T17:23:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.400+0000 7fbae5d9b640 10 monclient: get_auth_request con 0x7fbac8039430 auth_method 0 2024-05-21T17:23:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.404+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7fbadc015e30 con 0x7fbae80a4910 2024-05-21T17:23:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.404+0000 7fbad6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.404+0000 7fbad6ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:41.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.564+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbacc000ba0 con 0x7fbac8039430 2024-05-21T17:23:41.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbad6ffd640 1 -- 172.21.15.3:0/665201256 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33795 (unknown 2965378022 0 4195560622) 0x7fbacc000ba0 con 0x7fbac8039430 2024-05-21T17:23:41.572 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:23:41.573 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:23:41.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 >> v1:172.21.15.203:6812/343827626 conn(0x7fbac8039430 legacy=0x7fbac803b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbaec90f640 10 monclient: shutdown 2024-05-21T17:23:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbaec90f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 >> v1:172.21.15.3:6790/0 conn(0x7fbae80a4910 legacy=0x7fbae809e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 shutdown_connections 2024-05-21T17:23:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 >> 172.21.15.3:0/665201256 conn(0x7fbae80ab660 msgr2=0x7fbae80a3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 shutdown_connections 2024-05-21T17:23:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.568+0000 7fbaec90f640 1 -- 172.21.15.3:0/665201256 wait complete. 2024-05-21T17:23:41.588 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-05-21T17:23:41.093958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163352,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067688,"statfs":{"total":579820584960,"available":579653312512,"internally_reserved":0,"allocated":1236992,"data_stored":1439006,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003119"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.377367+0000","last_change":"2024-05-21T17:21:38.226793+0000","last_active":"2024-05-21T17:22:04.377367+0000","last_peered":"2024-05-21T17:22:04.377367+0000","last_clean":"2024-05-21T17:22:04.377367+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:22:04.377367+0000","last_undegraded":"2024-05-21T17:22:04.377367+0000","last_fullsized":"2024-05-21T17:22:04.377367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T21:23:30.360529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.371254+0000","last_change":"2024-05-21T17:21:38.830568+0000","last_active":"2024-05-21T17:22:04.371254+0000","last_peered":"2024-05-21T17:22:04.371254+0000","last_clean":"2024-05-21T17:22:04.371254+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:22:04.371254+0000","last_undegraded":"2024-05-21T17:22:04.371254+0000","last_fullsized":"2024-05-21T17:22:04.371254+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T02:04:57.235930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.731426+0000","last_change":"2024-05-21T17:21:38.216293+0000","last_active":"2024-05-21T17:22:04.731426+0000","last_peered":"2024-05-21T17:22:04.731426+0000","last_clean":"2024-05-21T17:22:04.731426+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:22:04.731426+0000","last_undegraded":"2024-05-21T17:22:04.731426+0000","last_fullsized":"2024-05-21T17:22:04.731426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:53:31.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.374521+0000","last_change":"2024-05-21T17:21:38.219369+0000","last_active":"2024-05-21T17:22:04.374521+0000","last_peered":"2024-05-21T17:22:04.374521+0000","last_clean":"2024-05-21T17:22:04.374521+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:22:04.374521+0000","last_undegraded":"2024-05-21T17:22:04.374521+0000","last_fullsized":"2024-05-21T17:22:04.374521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:24:49.428951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'72","reported_seq":114,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373702+0000","last_change":"2024-05-21T17:21:38.221127+0000","last_active":"2024-05-21T17:22:04.373702+0000","last_peered":"2024-05-21T17:22:04.373702+0000","last_clean":"2024-05-21T17:22:04.373702+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:22:04.373702+0000","last_undegraded":"2024-05-21T17:22:04.373702+0000","last_fullsized":"2024-05-21T17:22:04.373702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:12:45.986667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.370522+0000","last_change":"2024-05-21T17:21:38.219803+0000","last_active":"2024-05-21T17:22:04.370522+0000","last_peered":"2024-05-21T17:22:04.370522+0000","last_clean":"2024-05-21T17:22:04.370522+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:22:04.370522+0000","last_undegraded":"2024-05-21T17:22:04.370522+0000","last_fullsized":"2024-05-21T17:22:04.370522+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:52:49.624765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.729732+0000","last_change":"2024-05-21T17:21:38.829468+0000","last_active":"2024-05-21T17:22:04.729732+0000","last_peered":"2024-05-21T17:22:04.729732+0000","last_clean":"2024-05-21T17:22:04.729732+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:22:04.729732+0000","last_undegraded":"2024-05-21T17:22:04.729732+0000","last_fullsized":"2024-05-21T17:22:04.729732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:07:14.487479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373207+0000","last_change":"2024-05-21T17:21:38.830884+0000","last_active":"2024-05-21T17:22:04.373207+0000","last_peered":"2024-05-21T17:22:04.373207+0000","last_clean":"2024-05-21T17:22:04.373207+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:22:04.373207+0000","last_undegraded":"2024-05-21T17:22:04.373207+0000","last_fullsized":"2024-05-21T17:22:04.373207+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:36:41.643111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373983+0000","last_change":"2024-05-21T17:21:38.829272+0000","last_active":"2024-05-21T17:22:04.373983+0000","last_peered":"2024-05-21T17:22:04.373983+0000","last_clean":"2024-05-21T17:22:04.373983+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:22:04.373983+0000","last_undegraded":"2024-05-21T17:22:04.373983+0000","last_fullsized":"2024-05-21T17:22:04.373983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:09:03.800931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640282,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640283,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640284,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:23:41.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient: get_monmap_and_config 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient: build_initial_monmap 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient: monmap: 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:41.705482+0000 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:41.705482+0000 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:41.705 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 1 Processor -- start 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 1 -- start start 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient: init 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient: _add_conns ranks=[0,1,8,3,7,6,5,4,2] 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient(hunting): picked mon.noname-a con 0x7f31f80a4730 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient(hunting): picked mon.noname-f con 0x7f31f80a5d40 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient(hunting): picked mon.noname-e con 0x7f31f813f2f0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f80a6f50 con 0x7f31f80a4730 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f80a2960 con 0x7f31f80a5d40 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f80a2c40 con 0x7f31f813f2f0 2024-05-21T17:23:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient(hunting): authenticate will time out at 1088.862900s 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f6ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f31f80a4730 0x7f31f80a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55122/0 (socket says 172.21.15.3:55122) 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f6ffd640 1 -- 172.21.15.3:0/3122156576 learned_addr learned my addr 172.21.15.3:0/3122156576 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026170125 0 0) 0x7f31f80a6f50 con 0x7f31f80a4730 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient(hunting): my global_id is 4725 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31e0003530 con 0x7f31f80a4730 2024-05-21T17:23:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103632640 0 0) 0x7f31f80a2c40 con 0x7f31f813f2f0 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient(hunting): my global_id is 4724 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31f80a6f50 con 0x7f31f813f2f0 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2931964511 0 0) 0x7f31e0003530 con 0x7f31f80a4730 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 >> v1:172.21.15.3:6793/0 conn(0x7f31f813f2f0 legacy=0x7f31f81416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 >> v1:172.21.15.203:6789/0 conn(0x7f31f80a5d40 legacy=0x7f31f813dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 monclient: found mon.noname-a 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f80a5220 con 0x7f31f80a4730 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: _finish_auth 0 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f31e8001840 con 0x7f31f80a4730 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: dump: 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 5 monclient: authenticate success, global_id 4725 2024-05-21T17:23:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31e8001a60 con 0x7f31f80a4730 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 1 -- 172.21.15.3:0/3122156576 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f31e8001d50 con 0x7f31f80a4730 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31f5ffb640 10 monclient: dump: 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.700+0000 7f31fd40d640 4 set_mon_vals no callback set 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: shutdown 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- 172.21.15.3:0/3122156576 >> v1:172.21.15.3:6789/0 conn(0x7f31f80a4730 legacy=0x7f31f80a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- 172.21.15.3:0/3122156576 shutdown_connections 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- 172.21.15.3:0/3122156576 >> 172.21.15.3:0/3122156576 conn(0x7f31f80ab610 msgr2=0x7f31f80a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- 172.21.15.3:0/3122156576 shutdown_connections 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- 172.21.15.3:0/3122156576 wait complete. 2024-05-21T17:23:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: build_initial_monmap 2024-05-21T17:23:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: monmap: 2024-05-21T17:23:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 Processor -- start 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- start start 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: init 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: _add_conns ranks=[5,3,2,1,0,6,4,8,7] 2024-05-21T17:23:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient(hunting): picked mon.noname-h con 0x7f31f80a4730 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient(hunting): picked mon.noname-g con 0x7f31f80a5d40 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient(hunting): picked mon.noname-b con 0x7f31f813f2f0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f81389f0 con 0x7f31f813f2f0 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f80a1d40 con 0x7f31f80a5d40 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f80a13c0 con 0x7f31f80a4730 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient(hunting): authenticate will time out at 1088.865825s 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31f77fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f31f813f2f0 0x7f31f81d5d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48954/0 (socket says 172.21.15.3:48954) 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31f77fe640 1 -- 172.21.15.3:0/2138396633 learned_addr learned my addr 172.21.15.3:0/2138396633 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2391315503 0 0) 0x7f31f81389f0 con 0x7f31f813f2f0 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient(hunting): my global_id is 134219 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31e4003530 con 0x7f31f813f2f0 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2637365566 0 0) 0x7f31e4003530 con 0x7f31f813f2f0 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31f81389f0 con 0x7f31f813f2f0 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f31f0001d40 con 0x7f31f813f2f0 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient(hunting): dump: 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2753152602 0 0) 0x7f31f81389f0 con 0x7f31f813f2f0 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 >> v1:172.21.15.203:6791/0 conn(0x7f31f80a4730 legacy=0x7f31f8137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 >> v1:172.21.15.203:6790/0 conn(0x7f31f80a5d40 legacy=0x7f31f81382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 monclient: found mon.b 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f8139e30 con 0x7f31f813f2f0 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient: _finish_auth 0 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 5 monclient: authenticate success, global_id 134219 2024-05-21T17:23:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: _renew_subs 2024-05-21T17:23:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31f8139a70 con 0x7f31f813f2f0 2024-05-21T17:23:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: _renew_subs 2024-05-21T17:23:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f31f8138f40 con 0x7f31f813f2f0 2024-05-21T17:23:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31f0002000 con 0x7f31f813f2f0 2024-05-21T17:23:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f31f0003f70 con 0x7f31f813f2f0 2024-05-21T17:23:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31f57fa640 4 set_mon_vals no callback set 2024-05-21T17:23:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 10 monclient: dump: 2024-05-21T17:23:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:41.731 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:41.731 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:41.731 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:41.731 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:41.731 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:41.731 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:41.732 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d8005090 con 0x7f31f813f2f0 2024-05-21T17:23:41.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f31f0004110 con 0x7f31f813f2f0 2024-05-21T17:23:41.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.704+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f31f0045e00 con 0x7f31f813f2f0 2024-05-21T17:23:41.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.708+0000 7f31f6ffd640 10 monclient: get_auth_request con 0x7f31e4038d00 auth_method 0 2024-05-21T17:23:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.712+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f31f0015ec0 con 0x7f31f813f2f0 2024-05-21T17:23:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.712+0000 7f31df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.712+0000 7f31df7fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:41.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.864+0000 7f31fd40d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:23:41.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.864+0000 7f31fd40d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:23:41.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.864+0000 7f31fd40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:41.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.864+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31d8005380 con 0x7f31f813f2f0 2024-05-21T17:23:41.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31df7fe640 1 -- 172.21.15.3:0/2138396633 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+10763 (unknown 898900276 0 3622989622) 0x7f31f00157c0 con 0x7f31f813f2f0 2024-05-21T17:23:41.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-21T17:23:41.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31df7fe640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:23:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:23:41.872 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":21,"fsid":"4db3b12d-01c4-4d6c-b29b-1648ee37f87f","created":"2024-05-21T17:21:26.465014+0000","modified":"2024-05-21T17:22:04.359536+0000","last_up_change":"2024-05-21T17:21:35.154389+0000","last_in_change":"2024-05-21T17:21:28.263976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-21T17:21:36.581878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-21T17:21:37.102413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7dfaea0b-2159-4f6e-86a6-da6637052771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6800","nonce":857120552}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6801","nonce":857120552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6803","nonce":857120552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6802","nonce":857120552}]},"public_addr":"172.21.15.3:6800/857120552","cluster_addr":"172.21.15.3:6801/857120552","heartbeat_back_addr":"172.21.15.3:6803/857120552","heartbeat_front_addr":"172.21.15.3:6802/857120552","state":["exists","up"]},{"osd":1,"uuid":"28d053ac-034d-43ed-a83e-9ee62a426b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6808","nonce":4137544047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6809","nonce":4137544047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6811","nonce":4137544047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6810","nonce":4137544047}]},"public_addr":"172.21.15.3:6808/4137544047","cluster_addr":"172.21.15.3:6809/4137544047","heartbeat_back_addr":"172.21.15.3:6811/4137544047","heartbeat_front_addr":"172.21.15.3:6810/4137544047","state":["exists","up"]},{"osd":2,"uuid":"9de90871-f66b-4a85-ac48-772541c5b23d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6804","nonce":517111994}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6805","nonce":517111994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6807","nonce":517111994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.3:6806","nonce":517111994}]},"public_addr":"172.21.15.3:6804/517111994","cluster_addr":"172.21.15.3:6805/517111994","heartbeat_back_addr":"172.21.15.3:6807/517111994","heartbeat_front_addr":"172.21.15.3:6806/517111994","state":["exists","up"]},{"osd":3,"uuid":"f8431d10-6318-458b-9a94-51fc47273f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":1468009857}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1468009857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1468009857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":1468009857}]},"public_addr":"172.21.15.203:6804/1468009857","cluster_addr":"172.21.15.203:6805/1468009857","heartbeat_back_addr":"172.21.15.203:6807/1468009857","heartbeat_front_addr":"172.21.15.203:6806/1468009857","state":["exists","up"]},{"osd":4,"uuid":"028748a9-3ca4-4730-960e-8913c9504a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":4020495055}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":4020495055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":4020495055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":4020495055}]},"public_addr":"172.21.15.203:6800/4020495055","cluster_addr":"172.21.15.203:6801/4020495055","heartbeat_back_addr":"172.21.15.203:6803/4020495055","heartbeat_front_addr":"172.21.15.203:6802/4020495055","state":["exists","up"]},{"osd":5,"uuid":"87412b84-97f1-4350-9034-87f8f33da44a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":2732955435}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":2732955435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":2732955435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":2732955435}]},"public_addr":"172.21.15.203:6808/2732955435","cluster_addr":"172.21.15.203:6809/2732955435","heartbeat_back_addr":"172.21.15.203:6811/2732955435","heartbeat_front_addr":"172.21.15.203:6810/2732955435","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.740995+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.776640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.734901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.745181+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.676894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T17:21:33.827672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T17:23:41.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 >> v1:172.21.15.203:6812/343827626 conn(0x7f31e4038d00 legacy=0x7f31e403b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31fd40d640 10 monclient: shutdown 2024-05-21T17:23:41.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31fd40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:41.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 >> v1:172.21.15.3:6790/0 conn(0x7f31f813f2f0 legacy=0x7f31f81d5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:41.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 shutdown_connections 2024-05-21T17:23:41.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 >> 172.21.15.3:0/2138396633 conn(0x7f31f80ab610 msgr2=0x7f31f8135ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:41.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 shutdown_connections 2024-05-21T17:23:41.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:41.868+0000 7f31fd40d640 1 -- 172.21.15.3:0/2138396633 wait complete. 2024-05-21T17:23:42.889 INFO:tasks.ceph:Scrubbing osd.0 2024-05-21T17:23:42.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-21T17:23:43.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient: get_monmap_and_config 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient: monmap: 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:43.033187+0000 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:43.033187+0000 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 1 Processor -- start 2024-05-21T17:23:43.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 1 -- start start 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient: init 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient: _add_conns ranks=[2,7,1,4,8,6,3,0,5] 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient(hunting): picked mon.noname-b con 0x7fca280a3880 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient(hunting): picked mon.noname-i con 0x7fca280683f0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient(hunting): picked mon.noname-f con 0x7fca280a4370 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2809f900 con 0x7fca280a4370 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2809f3e0 con 0x7fca280a3880 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2806ced0 con 0x7fca280683f0 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient(hunting): authenticate will time out at 1090.190943s 2024-05-21T17:23:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca277fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fca280a3880 0x7fca280a3c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48968/0 (socket says 172.21.15.3:48968) 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca277fe640 1 -- 172.21.15.3:0/1412802000 learned_addr learned my addr 172.21.15.3:0/1412802000 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3899198533 0 0) 0x7fca2809f3e0 con 0x7fca280a3880 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient(hunting): my global_id is 134228 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca10003530 con 0x7fca280a3880 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 314652174 0 0) 0x7fca10003530 con 0x7fca280a3880 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 >> v1:172.21.15.203:6792/0 conn(0x7fca280683f0 legacy=0x7fca2813bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 >> v1:172.21.15.203:6789/0 conn(0x7fca280a4370 legacy=0x7fca2813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 monclient: found mon.noname-b 2024-05-21T17:23:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca28068a80 con 0x7fca280a3880 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 5 monclient: authenticate success, global_id 134228 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca18001840 con 0x7fca280a3880 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: dump: 2024-05-21T17:23:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca18001a60 con 0x7fca280a3880 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca2e1a0640 4 set_mon_vals no callback set 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 1 -- 172.21.15.3:0/1412802000 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca18004bf0 con 0x7fca280a3880 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.028+0000 7fca267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca267fc640 10 monclient: dump: 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: shutdown 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- 172.21.15.3:0/1412802000 >> v1:172.21.15.3:6790/0 conn(0x7fca280a3880 legacy=0x7fca280a3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- 172.21.15.3:0/1412802000 shutdown_connections 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- 172.21.15.3:0/1412802000 >> 172.21.15.3:0/1412802000 conn(0x7fca280a4f70 msgr2=0x7fca2806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- 172.21.15.3:0/1412802000 shutdown_connections 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- 172.21.15.3:0/1412802000 wait complete. 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: monmap: 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 Processor -- start 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- start start 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: init 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: _add_conns ranks=[2,6,0,1,7,3,8,4,5] 2024-05-21T17:23:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient(hunting): picked mon.noname-b con 0x7fca280683f0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient(hunting): picked mon.noname-d con 0x7fca280a3880 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient(hunting): picked mon.noname-a con 0x7fca280a4370 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2813e180 con 0x7fca280a4370 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2813bf20 con 0x7fca280683f0 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca28068e00 con 0x7fca280a3880 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient(hunting): authenticate will time out at 1090.194234s 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca277fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fca280683f0 0x7fca2813b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48982/0 (socket says 172.21.15.3:48982) 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca277fe640 1 -- 172.21.15.3:0/2450217363 learned_addr learned my addr 172.21.15.3:0/2450217363 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2850988267 0 0) 0x7fca2813bf20 con 0x7fca280683f0 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): my global_id is 134237 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca14003530 con 0x7fca280683f0 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 762389451 0 0) 0x7fca2809f900 con 0x7fca280a4370 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): my global_id is 4734 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca2813bf20 con 0x7fca280a4370 2024-05-21T17:23:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3420421759 0 0) 0x7fca28068e00 con 0x7fca280a3880 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): my global_id is 4524 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca2809f900 con 0x7fca280a3880 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3369928559 0 0) 0x7fca14003530 con 0x7fca280683f0 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca28068e00 con 0x7fca280683f0 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca18004b30 con 0x7fca280683f0 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): dump: 2024-05-21T17:23:43.053 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1108831768 0 0) 0x7fca2809f900 con 0x7fca280a3880 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca14003530 con 0x7fca280a3880 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3349483176 0 0) 0x7fca2813bf20 con 0x7fca280a4370 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca2809f900 con 0x7fca280a4370 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca1c0018b0 con 0x7fca280a3880 2024-05-21T17:23:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): dump: 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca20001cc0 con 0x7fca280a4370 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient(hunting): dump: 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3069697355 0 0) 0x7fca14003530 con 0x7fca280a3880 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 >> v1:172.21.15.3:6790/0 conn(0x7fca280683f0 legacy=0x7fca2813b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 >> v1:172.21.15.3:6789/0 conn(0x7fca280a4370 legacy=0x7fca281d3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 monclient: found mon.d 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca28073ce0 con 0x7fca280a3880 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 5 monclient: authenticate success, global_id 4524 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: _renew_subs 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca2813da80 con 0x7fca280a3880 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca1c001a50 con 0x7fca280a3880 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: _renew_subs 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fca28140720 con 0x7fca280a3880 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fca1c003a40 con 0x7fca280a3880 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca0ffff640 10 monclient: dump: 2024-05-21T17:23:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.032+0000 7fca25ffb640 4 set_mon_vals no callback set 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca2e1a0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fca2806bf50 con 0x7fca280a3880 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fca1c00ebc0 con 0x7fca280a3880 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fca1c044770 con 0x7fca280a3880 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6800/857120552 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca14043e30 con 0x7fca140401c0 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== mon.6 v1:172.21.15.3:6792/0 9 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (unknown 3512546819 0 0) 0x7fca1c044b80 con 0x7fca280a3880 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca0ffff640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca277fe640 10 monclient: get_auth_request con 0x7fca140395f0 auth_method 0 2024-05-21T17:23:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca27fff640 10 monclient: get_auth_request con 0x7fca140401c0 auth_method 0 2024-05-21T17:23:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.036+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== osd.0 v1:172.21.15.3:6800/857120552 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca14043e30 con 0x7fca140401c0 2024-05-21T17:23:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 --> v1:172.21.15.3:6800/857120552 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fca28140a20 con 0x7fca140401c0 2024-05-21T17:23:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca0ffff640 1 -- 172.21.15.3:0/2450217363 <== osd.0 v1:172.21.15.3:6800/857120552 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fca28140a20 con 0x7fca140401c0 2024-05-21T17:23:43.066 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:43.066 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-21T17:23:43.066 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:43.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 >> v1:172.21.15.3:6800/857120552 conn(0x7fca140401c0 legacy=0x7fca140425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 >> v1:172.21.15.203:6812/343827626 conn(0x7fca140395f0 legacy=0x7fca1403bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 10 monclient: shutdown 2024-05-21T17:23:43.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 >> v1:172.21.15.3:6792/0 conn(0x7fca280a3880 legacy=0x7fca2813b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca27fff640 1 -- 172.21.15.3:0/2450217363 reap_dead start 2024-05-21T17:23:43.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 shutdown_connections 2024-05-21T17:23:43.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 >> 172.21.15.3:0/2450217363 conn(0x7fca280a4f70 msgr2=0x7fca280a6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 shutdown_connections 2024-05-21T17:23:43.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.048+0000 7fca2e1a0640 1 -- 172.21.15.3:0/2450217363 wait complete. 2024-05-21T17:23:43.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient: get_monmap_and_config 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient: monmap: 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:43.188368+0000 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:43.188368+0000 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 Processor -- start 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- start start 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient: init 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient: _add_conns ranks=[1,2,8,5,6,0,4,7,3] 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient(hunting): picked mon.noname-f con 0x7feafc0a48c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient(hunting): picked mon.noname-b con 0x7feafc0a4ca0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient(hunting): picked mon.noname-e con 0x7feafc13f210 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:43.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feafc0a88a0 con 0x7feafc0a48c0 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feafc0a6550 con 0x7feafc0a4ca0 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feafc0a28c0 con 0x7feafc13f210 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient(hunting): authenticate will time out at 1090.345986s 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafaffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7feafc0a4ca0 0x7feafc13daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48988/0 (socket says 172.21.15.3:48988) 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafaffd640 1 -- 172.21.15.3:0/3454800054 learned_addr learned my addr 172.21.15.3:0/3454800054 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafbfff640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7feafc13f210 0x7feafc141600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54820/0 (socket says 172.21.15.3:54820) 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3219079104 0 0) 0x7feafc0a6550 con 0x7feafc0a4ca0 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient(hunting): my global_id is 134246 2024-05-21T17:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feae4003530 con 0x7feafc0a4ca0 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 546990883 0 0) 0x7feafc0a28c0 con 0x7feafc13f210 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient(hunting): my global_id is 4733 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feafc0a6550 con 0x7feafc13f210 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 338538738 0 0) 0x7feae4003530 con 0x7feafc0a4ca0 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 >> v1:172.21.15.3:6793/0 conn(0x7feafc13f210 legacy=0x7feafc141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 >> v1:172.21.15.203:6789/0 conn(0x7feafc0a48c0 legacy=0x7feafc0a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 monclient: found mon.noname-b 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feafc0a2ba0 con 0x7feafc0a4ca0 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feaf0001840 con 0x7feafc0a4ca0 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 5 monclient: authenticate success, global_id 134246 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: dump: 2024-05-21T17:23:43.199 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feaf0001a60 con 0x7feafc0a4ca0 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 1 -- 172.21.15.3:0/3454800054 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feaf0001d50 con 0x7feafc0a4ca0 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feafa7fc640 10 monclient: dump: 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 4 set_mon_vals no callback set 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 10 monclient: shutdown 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- 172.21.15.3:0/3454800054 >> v1:172.21.15.3:6790/0 conn(0x7feafc0a4ca0 legacy=0x7feafc13daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- 172.21.15.3:0/3454800054 shutdown_connections 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- 172.21.15.3:0/3454800054 >> 172.21.15.3:0/3454800054 conn(0x7feafc0ab5b0 msgr2=0x7feafc0a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- 172.21.15.3:0/3454800054 shutdown_connections 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.184+0000 7feb01ba2640 1 -- 172.21.15.3:0/3454800054 wait complete. 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: monmap: 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:43.203 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.204 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 1 Processor -- start 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 1 -- start start 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: init 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: _add_conns ranks=[2,8,6,3,4,5,0,7,1] 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient(hunting): picked mon.noname-b con 0x7feafc0a48c0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient(hunting): picked mon.noname-e con 0x7feafc0a4ca0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient(hunting): picked mon.noname-d con 0x7feafc13f210 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feafc13b990 con 0x7feafc0a48c0 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feafc13c020 con 0x7feafc13f210 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feafc13d4d0 con 0x7feafc0a4ca0 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient(hunting): authenticate will time out at 1090.349167s 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feafb7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7feafc0a48c0 0x7feafc138a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48998/0 (socket says 172.21.15.3:48998) 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feafb7fe640 1 -- 172.21.15.3:0/2972954074 learned_addr learned my addr 172.21.15.3:0/2972954074 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3345916339 0 0) 0x7feafc13b990 con 0x7feafc0a48c0 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): my global_id is 134255 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feae8003530 con 0x7feafc0a48c0 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533010243 0 0) 0x7feafc13d4d0 con 0x7feafc0a4ca0 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): my global_id is 4742 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feafc13b990 con 0x7feafc0a4ca0 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780677721 0 0) 0x7feafc13c020 con 0x7feafc13f210 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): my global_id is 4533 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feafc13d4d0 con 0x7feafc13f210 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1941736104 0 0) 0x7feae8003530 con 0x7feafc0a48c0 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feafc13c020 con 0x7feafc0a48c0 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2693544340 0 0) 0x7feafc13b990 con 0x7feafc0a4ca0 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feae8003530 con 0x7feafc0a4ca0 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feaf4001e60 con 0x7feafc0a48c0 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): dump: 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feaf0001800 con 0x7feafc0a4ca0 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): dump: 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.210 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2172314268 0 0) 0x7feafc13d4d0 con 0x7feafc13f210 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feafc13b990 con 0x7feafc13f210 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feaec001770 con 0x7feafc13f210 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient(hunting): dump: 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3931635171 0 0) 0x7feafc13c020 con 0x7feafc0a48c0 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 >> v1:172.21.15.3:6793/0 conn(0x7feafc0a4ca0 legacy=0x7feafc1d2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 >> v1:172.21.15.3:6792/0 conn(0x7feafc13f210 legacy=0x7feafc1d5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 monclient: found mon.b 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feafc13d1f0 con 0x7feafc0a48c0 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 5 monclient: authenticate success, global_id 134255 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: _renew_subs 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feafc13bd30 con 0x7feafc0a48c0 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: _renew_subs 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feafc1391a0 con 0x7feafc0a48c0 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feaf4003c50 con 0x7feafc0a48c0 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7feaf4004140 con 0x7feafc0a48c0 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feaf9ffb640 4 set_mon_vals no callback set 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 10 monclient: dump: 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.214 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feadc005090 con 0x7feafc0a48c0 2024-05-21T17:23:43.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7feaf40042e0 con 0x7feafc0a48c0 2024-05-21T17:23:43.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7feaf4045ef0 con 0x7feafc0a48c0 2024-05-21T17:23:43.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.188+0000 7feafaffd640 10 monclient: get_auth_request con 0x7feae8039670 auth_method 0 2024-05-21T17:23:43.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.192+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7feaf4015ff0 con 0x7feafc0a48c0 2024-05-21T17:23:43.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.192+0000 7feae3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:43.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.192+0000 7feae3fff640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:43.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.356+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7feadc000ba0 con 0x7feae8039670 2024-05-21T17:23:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.356+0000 7feae3fff640 1 -- 172.21.15.3:0/2972954074 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7feadc000ba0 con 0x7feae8039670 2024-05-21T17:23:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 0 to deep-scrub 2024-05-21T17:23:43.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.360+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 >> v1:172.21.15.203:6812/343827626 conn(0x7feae8039670 legacy=0x7feae803bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.360+0000 7feb01ba2640 10 monclient: shutdown 2024-05-21T17:23:43.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.360+0000 7feb01ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.360+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 >> v1:172.21.15.3:6790/0 conn(0x7feafc0a48c0 legacy=0x7feafc138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.360+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 shutdown_connections 2024-05-21T17:23:43.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.360+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 >> 172.21.15.3:0/2972954074 conn(0x7feafc0ab5b0 msgr2=0x7feafc0a3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.360+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 shutdown_connections 2024-05-21T17:23:43.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.360+0000 7feb01ba2640 1 -- 172.21.15.3:0/2972954074 wait complete. 2024-05-21T17:23:43.381 INFO:tasks.ceph:Scrubbing osd.1 2024-05-21T17:23:43.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-21T17:23:43.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient: get_monmap_and_config 2024-05-21T17:23:43.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient: monmap: 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:43.509938+0000 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:43.509938+0000 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.510 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.510 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.510 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.510 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 1 Processor -- start 2024-05-21T17:23:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 1 -- start start 2024-05-21T17:23:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient: init 2024-05-21T17:23:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient: _add_conns ranks=[5,1,7,2,4,3,6,8,0] 2024-05-21T17:23:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient(hunting): picked mon.noname-h con 0x7fc2d4135980 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient(hunting): picked mon.noname-f con 0x7fc2d409db30 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient(hunting): picked mon.noname-i con 0x7fc2d413f320 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d4136470 con 0x7fc2d409db30 2024-05-21T17:23:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d40743d0 con 0x7fc2d4135980 2024-05-21T17:23:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d4068380 con 0x7fc2d413f320 2024-05-21T17:23:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2da070640 10 monclient(hunting): authenticate will time out at 1090.667364s 2024-05-21T17:23:43.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2d3fff640 1 --1- >> v1:172.21.15.203:6792/0 conn(0x7fc2d413f320 0x7fc2d4141710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6792/0 says I am v1:172.21.15.3:48242/0 (socket says 172.21.15.3:48242) 2024-05-21T17:23:43.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.504+0000 7fc2d3fff640 1 -- 172.21.15.3:0/2741517718 learned_addr learned my addr 172.21.15.3:0/2741517718 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141788107 0 0) 0x7fc2d40743d0 con 0x7fc2d4135980 2024-05-21T17:23:43.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): my global_id is 4757 2024-05-21T17:23:43.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2bc003530 con 0x7fc2d4135980 2024-05-21T17:23:43.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1011636996 0 0) 0x7fc2d4068380 con 0x7fc2d413f320 2024-05-21T17:23:43.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): my global_id is 4759 2024-05-21T17:23:43.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d40743d0 con 0x7fc2d413f320 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2264519803 0 0) 0x7fc2d4136470 con 0x7fc2d409db30 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient(hunting): my global_id is 134227 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d4068380 con 0x7fc2d409db30 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2062913070 0 0) 0x7fc2d40743d0 con 0x7fc2d413f320 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 >> v1:172.21.15.203:6791/0 conn(0x7fc2d4135980 legacy=0x7fc2d4135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 >> v1:172.21.15.203:6789/0 conn(0x7fc2d409db30 legacy=0x7fc2d413dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 monclient: found mon.noname-i 2024-05-21T17:23:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.203:6792/0 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 --> v1:172.21.15.203:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d409ce70 con 0x7fc2d413f320 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc2cc001ba0 con 0x7fc2d413f320 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: dump: 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.527 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 5 monclient: authenticate success, global_id 4759 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 <== mon.7 v1:172.21.15.203:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2cc001dc0 con 0x7fc2d413f320 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 4 set_mon_vals no callback set 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 1 -- 172.21.15.3:0/2741517718 <== mon.7 v1:172.21.15.203:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc2cc004f70 con 0x7fc2d413f320 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d27fc640 10 monclient: dump: 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient: shutdown 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- 172.21.15.3:0/2741517718 >> v1:172.21.15.203:6792/0 conn(0x7fc2d413f320 legacy=0x7fc2d4141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- 172.21.15.3:0/2741517718 shutdown_connections 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- 172.21.15.3:0/2741517718 >> 172.21.15.3:0/2741517718 conn(0x7fc2d40a4f70 msgr2=0x7fc2d406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- 172.21.15.3:0/2741517718 shutdown_connections 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- 172.21.15.3:0/2741517718 wait complete. 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient: monmap: 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.531 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 Processor -- start 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- start start 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient: init 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient: _add_conns ranks=[4,8,7,5,3,2,1,6,0] 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient(hunting): picked mon.noname-c con 0x7fc2d409db30 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient(hunting): picked mon.noname-e con 0x7fc2d4135980 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient(hunting): picked mon.noname-i con 0x7fc2d413f320 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d4134890 con 0x7fc2d409db30 2024-05-21T17:23:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d409e510 con 0x7fc2d413f320 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d4136a90 con 0x7fc2d4135980 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2da070640 10 monclient(hunting): authenticate will time out at 1090.671779s 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d37fe640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fc2d409db30 0x7fc2d412b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:56924/0 (socket says 172.21.15.3:56924) 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d37fe640 1 -- 172.21.15.3:0/3270826098 learned_addr learned my addr 172.21.15.3:0/3270826098 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.508+0000 7fc2d2ffd640 1 --1- 172.21.15.3:0/3270826098 >> v1:172.21.15.3:6793/0 conn(0x7fc2d4135980 0x7fc2d41d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54838/0 (socket says 172.21.15.3:54838) 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887645042 0 0) 0x7fc2d4134890 con 0x7fc2d409db30 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): my global_id is 4702 2024-05-21T17:23:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2c0003530 con 0x7fc2d409db30 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3257644961 0 0) 0x7fc2d4136a90 con 0x7fc2d4135980 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): my global_id is 4751 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d4134890 con 0x7fc2d4135980 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3612988014 0 0) 0x7fc2d409e510 con 0x7fc2d413f320 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): my global_id is 4768 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d4136a90 con 0x7fc2d413f320 2024-05-21T17:23:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2699513175 0 0) 0x7fc2c0003530 con 0x7fc2d409db30 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2d409e510 con 0x7fc2d409db30 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc2c8001680 con 0x7fc2d409db30 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): dump: 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3465817045 0 0) 0x7fc2d4134890 con 0x7fc2d4135980 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2c0003530 con 0x7fc2d4135980 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc2c40019a0 con 0x7fc2d4135980 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:43.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient(hunting): dump: 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3806296563 0 0) 0x7fc2d409e510 con 0x7fc2d409db30 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 >> v1:172.21.15.3:6793/0 conn(0x7fc2d4135980 legacy=0x7fc2d41d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 >> v1:172.21.15.203:6792/0 conn(0x7fc2d413f320 legacy=0x7fc2d41d5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 monclient: found mon.c 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d4136d70 con 0x7fc2d409db30 2024-05-21T17:23:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 5 monclient: authenticate success, global_id 4702 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 10 monclient: _renew_subs 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d41366d0 con 0x7fc2d409db30 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 10 monclient: _renew_subs 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fc2d4134c30 con 0x7fc2d409db30 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2c80038b0 con 0x7fc2d409db30 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fc2c8003ea0 con 0x7fc2d409db30 2024-05-21T17:23:43.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2d17fa640 4 set_mon_vals no callback set 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: dump: 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.540 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc2b4000f90 con 0x7fc2d409db30 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fc2c8004040 con 0x7fc2d409db30 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fc2c8045be0 con 0x7fc2d409db30 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6808/4137544047 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2c0043c10 con 0x7fc2c003ffa0 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (unknown 3512546819 0 0) 0x7fc2c8045ff0 con 0x7fc2d409db30 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-05-21T17:23:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2d3fff640 10 monclient: get_auth_request con 0x7fc2c003ffa0 auth_method 0 2024-05-21T17:23:43.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2d2ffd640 10 monclient: get_auth_request con 0x7fc2c0039460 auth_method 0 2024-05-21T17:23:43.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.512+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== osd.1 v1:172.21.15.3:6808/4137544047 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc2c0043c10 con 0x7fc2c003ffa0 2024-05-21T17:23:43.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 --> v1:172.21.15.3:6808/4137544047 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc2b4002ce0 con 0x7fc2c003ffa0 2024-05-21T17:23:43.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2bbfff640 1 -- 172.21.15.3:0/3270826098 <== osd.1 v1:172.21.15.3:6808/4137544047 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fc2b4002ce0 con 0x7fc2c003ffa0 2024-05-21T17:23:43.543 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:43.543 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-21T17:23:43.543 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 >> v1:172.21.15.3:6808/4137544047 conn(0x7fc2c003ffa0 legacy=0x7fc2c00423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 >> v1:172.21.15.203:6812/343827626 conn(0x7fc2c0039460 legacy=0x7fc2c003b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 10 monclient: shutdown 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 >> v1:172.21.15.3:6791/0 conn(0x7fc2d409db30 legacy=0x7fc2d412b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2d3fff640 1 -- 172.21.15.3:0/3270826098 reap_dead start 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 shutdown_connections 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 >> 172.21.15.3:0/3270826098 conn(0x7fc2d40a4f70 msgr2=0x7fc2d40a6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 shutdown_connections 2024-05-21T17:23:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.528+0000 7fc2da070640 1 -- 172.21.15.3:0/3270826098 wait complete. 2024-05-21T17:23:43.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-21T17:23:43.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.648+0000 7f6dbb1c9640 10 monclient: get_monmap_and_config 2024-05-21T17:23:43.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.648+0000 7f6dbb1c9640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.648+0000 7f6dbb1c9640 10 monclient: monmap: 2024-05-21T17:23:43.654 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.654 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:43.655881+0000 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:43.655881+0000 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.655 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.656 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.656 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 Processor -- start 2024-05-21T17:23:43.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- start start 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient: init 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient: _add_conns ranks=[8,5,0,1,7,4,2,3,6] 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient(hunting): picked mon.noname-e con 0x7f6db40a5bd0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient(hunting): picked mon.noname-h con 0x7f6db40a66c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient(hunting): picked mon.noname-a con 0x7f6db40a70c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db409e480 con 0x7f6db40a70c0 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db409cce0 con 0x7f6db40a66c0 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db409cfc0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient(hunting): authenticate will time out at 1090.813351s 2024-05-21T17:23:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db9766640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6db40a70c0 0x7f6db4141370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55132/0 (socket says 172.21.15.3:55132) 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db9766640 1 -- 172.21.15.3:0/3311991389 learned_addr learned my addr 172.21.15.3:0/3311991389 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db8f65640 1 --1- 172.21.15.3:0/3311991389 >> v1:172.21.15.3:6793/0 conn(0x7f6db40a5bd0 0x7f6db40a5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:54840/0 (socket says 172.21.15.3:54840) 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4246180258 0 0) 0x7f6db409e480 con 0x7f6db40a70c0 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient(hunting): my global_id is 4743 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d9c003530 con 0x7f6db40a70c0 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1270986461 0 0) 0x7f6db409cfc0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient(hunting): my global_id is 4760 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6db409e480 con 0x7f6db40a5bd0 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3276287755 0 0) 0x7f6db409e480 con 0x7f6db40a5bd0 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 >> v1:172.21.15.203:6791/0 conn(0x7f6db40a66c0 legacy=0x7f6db413dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 >> v1:172.21.15.3:6789/0 conn(0x7f6db40a70c0 legacy=0x7f6db4141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 monclient: found mon.noname-e 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db40742e0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6da4001820 con 0x7f6db40a5bd0 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: dump: 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 5 monclient: authenticate success, global_id 4760 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6da40019c0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 1 -- 172.21.15.3:0/3311991389 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6da4004b30 con 0x7f6db40a5bd0 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6db37fe640 10 monclient: dump: 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 4 set_mon_vals no callback set 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient: shutdown 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/3311991389 >> v1:172.21.15.3:6793/0 conn(0x7f6db40a5bd0 legacy=0x7f6db40a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/3311991389 shutdown_connections 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/3311991389 >> 172.21.15.3:0/3311991389 conn(0x7f6db40ab7d0 msgr2=0x7f6db40a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/3311991389 shutdown_connections 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/3311991389 wait complete. 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.652+0000 7f6dbb1c9640 10 monclient: monmap: 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 1 Processor -- start 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 1 -- start start 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: init 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: _add_conns ranks=[2,1,3,6,4,7,0,5,8] 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient(hunting): picked mon.noname-b con 0x7f6db40a5bd0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient(hunting): picked mon.noname-f con 0x7f6db40a66c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient(hunting): picked mon.noname-g con 0x7f6db40a70c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db4142250 con 0x7f6db40a66c0 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db4122220 con 0x7f6db40a5bd0 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db40a79f0 con 0x7f6db40a70c0 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient(hunting): authenticate will time out at 1090.817117s 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db8f65640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6db40a5bd0 0x7f6db413c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:49000/0 (socket says 172.21.15.3:49000) 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db8f65640 1 -- 172.21.15.3:0/1127980315 learned_addr learned my addr 172.21.15.3:0/1127980315 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 79230069 0 0) 0x7f6db4122220 con 0x7f6db40a5bd0 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient(hunting): my global_id is 134264 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6da0003530 con 0x7f6db40a5bd0 2024-05-21T17:23:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 668677240 0 0) 0x7f6da0003530 con 0x7f6db40a5bd0 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6db4122220 con 0x7f6db40a5bd0 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6da4001760 con 0x7f6db40a5bd0 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient(hunting): dump: 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1099548550 0 0) 0x7f6db4122220 con 0x7f6db40a5bd0 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 >> v1:172.21.15.203:6790/0 conn(0x7f6db40a70c0 legacy=0x7f6db41d5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 >> v1:172.21.15.203:6789/0 conn(0x7f6db40a66c0 legacy=0x7f6db41c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 monclient: found mon.b 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db4123740 con 0x7f6db40a5bd0 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 5 monclient: authenticate success, global_id 134264 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: _renew_subs 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6db4121fa0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: _renew_subs 2024-05-21T17:23:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6db41ca450 con 0x7f6db40a5bd0 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6da4004df0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6da40019c0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db2ffd640 4 set_mon_vals no callback set 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 10 monclient: dump: 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d94005090 con 0x7f6db40a5bd0 2024-05-21T17:23:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f6da40102c0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f6da4045cd0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.656+0000 7f6db3fff640 10 monclient: get_auth_request con 0x7f6da0038e50 auth_method 0 2024-05-21T17:23:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.660+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f6da4015df0 con 0x7f6db40a5bd0 2024-05-21T17:23:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.660+0000 7f6db17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:43.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.660+0000 7f6db17fa640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:43.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.816+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f6d94000ba0 con 0x7f6da0038e50 2024-05-21T17:23:43.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6db17fa640 1 -- 172.21.15.3:0/1127980315 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f6d94000ba0 con 0x7f6da0038e50 2024-05-21T17:23:43.823 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 1 to deep-scrub 2024-05-21T17:23:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 >> v1:172.21.15.203:6812/343827626 conn(0x7f6da0038e50 legacy=0x7f6da003b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6dbb1c9640 10 monclient: shutdown 2024-05-21T17:23:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6dbb1c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 >> v1:172.21.15.3:6790/0 conn(0x7f6db40a5bd0 legacy=0x7f6db413c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 shutdown_connections 2024-05-21T17:23:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 >> 172.21.15.3:0/1127980315 conn(0x7f6db40ab7d0 msgr2=0x7f6db40a0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 shutdown_connections 2024-05-21T17:23:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.820+0000 7f6dbb1c9640 1 -- 172.21.15.3:0/1127980315 wait complete. 2024-05-21T17:23:43.839 INFO:tasks.ceph:Scrubbing osd.2 2024-05-21T17:23:43.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-21T17:23:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient: get_monmap_and_config 2024-05-21T17:23:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient: monmap: 2024-05-21T17:23:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:43.965802+0000 2024-05-21T17:23:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:43.965802+0000 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.965 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 1 Processor -- start 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 1 -- start start 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient: init 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient: _add_conns ranks=[2,1,5,3,7,0,4,8,6] 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient(hunting): picked mon.noname-b con 0x7f6db80a38f0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient(hunting): picked mon.noname-f con 0x7f6db80a3cd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient(hunting): picked mon.noname-h con 0x7f6db813bb70 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db80743a0 con 0x7f6db80a3cd0 2024-05-21T17:23:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db809cdb0 con 0x7f6db80a38f0 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db809d1f0 con 0x7f6db813bb70 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.960+0000 7f6dbe45b640 10 monclient(hunting): authenticate will time out at 1091.123974s 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db7fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6db80a38f0 0x7f6db809fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:41088/0 (socket says 172.21.15.3:41088) 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db7fff640 1 -- 172.21.15.3:0/2725673350 learned_addr learned my addr 172.21.15.3:0/2725673350 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429979228 0 0) 0x7f6db809cdb0 con 0x7f6db80a38f0 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient(hunting): my global_id is 134273 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6da0003530 con 0x7f6db80a38f0 2024-05-21T17:23:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4163532634 0 0) 0x7f6da0003530 con 0x7f6db80a38f0 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 >> v1:172.21.15.203:6791/0 conn(0x7f6db813bb70 legacy=0x7f6db813df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 >> v1:172.21.15.203:6789/0 conn(0x7f6db80a3cd0 legacy=0x7f6db80a40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 monclient: found mon.noname-b 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db80a5380 con 0x7f6db80a38f0 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 5 monclient: authenticate success, global_id 134273 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6da8001840 con 0x7f6db80a38f0 2024-05-21T17:23:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: dump: 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6da8001a60 con 0x7f6db80a38f0 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 1 -- 172.21.15.3:0/2725673350 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6da8004bf0 con 0x7f6db80a38f0 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6db6ffd640 10 monclient: dump: 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.974 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 4 set_mon_vals no callback set 2024-05-21T17:23:43.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 10 monclient: shutdown 2024-05-21T17:23:43.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 1 -- 172.21.15.3:0/2725673350 >> v1:172.21.15.3:6790/0 conn(0x7f6db80a38f0 legacy=0x7f6db809fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 1 -- 172.21.15.3:0/2725673350 shutdown_connections 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 1 -- 172.21.15.3:0/2725673350 >> 172.21.15.3:0/2725673350 conn(0x7f6db80a84e0 msgr2=0x7f6db80a49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 1 -- 172.21.15.3:0/2725673350 shutdown_connections 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 1 -- 172.21.15.3:0/2725673350 wait complete. 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 10 monclient: build_initial_monmap 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 10 monclient: monmap: 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:43.979 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.964+0000 7f6dbe45b640 1 Processor -- start 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 1 -- start start 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient: init 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient: _add_conns ranks=[3,7,8,5,0,1,6,2,4] 2024-05-21T17:23:43.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient(hunting): picked mon.noname-g con 0x7f6db80a38f0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient(hunting): picked mon.noname-i con 0x7f6db80a3cd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient(hunting): picked mon.noname-e con 0x7f6db813bb70 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db8137880 con 0x7f6db80a38f0 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db8137b60 con 0x7f6db80a3cd0 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db806d2f0 con 0x7f6db813bb70 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient(hunting): authenticate will time out at 1091.128553s 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbc9f8640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f6db813bb70 0x7f6db81d3cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:55072/0 (socket says 172.21.15.3:55072) 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbc9f8640 1 -- 172.21.15.3:0/3726915816 learned_addr learned my addr 172.21.15.3:0/3726915816 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832167078 0 0) 0x7f6db806d2f0 con 0x7f6db813bb70 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): my global_id is 4769 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6da4003530 con 0x7f6db813bb70 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 613117653 0 0) 0x7f6da4003530 con 0x7f6db813bb70 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6db806d2f0 con 0x7f6db813bb70 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6db0001de0 con 0x7f6db813bb70 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): dump: 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.982 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059174141 0 0) 0x7f6db8137880 con 0x7f6db80a38f0 2024-05-21T17:23:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): my global_id is 4710 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6da4003530 con 0x7f6db80a38f0 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993381851 0 0) 0x7f6db8137b60 con 0x7f6db80a3cd0 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _negotiate 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient(hunting): my global_id is 4777 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6db8137880 con 0x7f6db80a3cd0 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3144954571 0 0) 0x7f6db806d2f0 con 0x7f6db813bb70 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 >> v1:172.21.15.203:6792/0 conn(0x7f6db80a3cd0 legacy=0x7f6db806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 >> v1:172.21.15.203:6790/0 conn(0x7f6db80a38f0 legacy=0x7f6db806c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: _finish_hunting 0 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 monclient: found mon.e 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db8136bc0 con 0x7f6db813bb70 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: _finish_auth 0 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: _check_auth_tickets 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 5 monclient: authenticate success, global_id 4769 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient: _renew_subs 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.3:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6db8074070 con 0x7f6db813bb70 2024-05-21T17:23:43.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient: _renew_subs 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.3:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6db8138260 con 0x7f6db813bb70 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6db0003c50 con 0x7f6db813bb70 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db67fc640 4 set_mon_vals no callback set 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f6db0004200 con 0x7f6db813bb70 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: dump: 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f6db0004420 con 0x7f6db813bb70 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.3:6793/0 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.3:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6d98000f90 con 0x7f6db813bb70 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f6db0045fa0 con 0x7f6db813bb70 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.3:6804/517111994 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6da4043670 con 0x7f6da403fa00 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== mon.8 v1:172.21.15.3:6793/0 9 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (unknown 3512546819 0 0) 0x7f6db0046330 con 0x7f6db813bb70 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db77fe640 10 monclient: get_auth_request con 0x7f6da403fa00 auth_method 0 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db7fff640 10 monclient: get_auth_request con 0x7f6da4038e00 auth_method 0 2024-05-21T17:23:43.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.968+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== osd.2 v1:172.21.15.3:6804/517111994 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6da4043670 con 0x7f6da403fa00 2024-05-21T17:23:43.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 --> v1:172.21.15.3:6804/517111994 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6d98002ce0 con 0x7f6da403fa00 2024-05-21T17:23:43.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6db4ff9640 1 -- 172.21.15.3:0/3726915816 <== osd.2 v1:172.21.15.3:6804/517111994 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f6d98002ce0 con 0x7f6da403fa00 2024-05-21T17:23:43.989 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:43.989 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-21T17:23:43.989 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:43.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 >> v1:172.21.15.3:6804/517111994 conn(0x7f6da403fa00 legacy=0x7f6da4041e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 >> v1:172.21.15.203:6812/343827626 conn(0x7f6da4038e00 legacy=0x7f6da403b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 10 monclient: shutdown 2024-05-21T17:23:43.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:43.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 >> v1:172.21.15.3:6793/0 conn(0x7f6db813bb70 legacy=0x7f6db81d3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:43.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbc9f8640 1 -- 172.21.15.3:0/3726915816 reap_dead start 2024-05-21T17:23:43.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 shutdown_connections 2024-05-21T17:23:43.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 >> 172.21.15.3:0/3726915816 conn(0x7f6db80a84e0 msgr2=0x7f6db806bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:43.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 shutdown_connections 2024-05-21T17:23:43.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:43.984+0000 7f6dbe45b640 1 -- 172.21.15.3:0/3726915816 wait complete. 2024-05-21T17:23:44.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-21T17:23:44.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient: get_monmap_and_config 2024-05-21T17:23:44.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient: build_initial_monmap 2024-05-21T17:23:44.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient: monmap: 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:44.113982+0000 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:44.113982+0000 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:44.113 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:44.114 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:44.114 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:44.114 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:44.114 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 1 Processor -- start 2024-05-21T17:23:44.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 1 -- start start 2024-05-21T17:23:44.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient: init 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient: _add_conns ranks=[0,6,4,1,3,7,5,2,8] 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient(hunting): picked mon.noname-a con 0x7f7be413d0e0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient(hunting): picked mon.noname-d con 0x7f7be409f8f0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient(hunting): picked mon.noname-c con 0x7f7be41436d0 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be409e1c0 con 0x7f7be413d0e0 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be409e670 con 0x7f7be41436d0 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be40a26d0 con 0x7f7be409f8f0 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7bebad0640 10 monclient(hunting): authenticate will time out at 1091.271433s 2024-05-21T17:23:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7be986c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7be413d0e0 0x7f7be413d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38546/0 (socket says 172.21.15.3:38546) 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7be986c640 1 -- 172.21.15.3:0/4146436531 learned_addr learned my addr 172.21.15.3:0/4146436531 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458817379 0 0) 0x7f7be409e1c0 con 0x7f7be413d0e0 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7be886a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.108+0000 7f7be886a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): my global_id is 4752 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bd0003530 con 0x7f7be413d0e0 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 20637130 0 0) 0x7f7be40a26d0 con 0x7f7be409f8f0 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): my global_id is 4542 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7be409e1c0 con 0x7f7be409f8f0 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702162954 0 0) 0x7f7be409e670 con 0x7f7be41436d0 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient(hunting): my global_id is 4711 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7be40a26d0 con 0x7f7be41436d0 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 896485270 0 0) 0x7f7be409e1c0 con 0x7f7be409f8f0 2024-05-21T17:23:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 >> v1:172.21.15.3:6791/0 conn(0x7f7be41436d0 legacy=0x7f7be4145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 >> v1:172.21.15.3:6789/0 conn(0x7f7be413d0e0 legacy=0x7f7be413d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: _finish_hunting 0 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 monclient: found mon.noname-d 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be409fcd0 con 0x7f7be409f8f0 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: _finish_auth 0 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: _check_auth_tickets 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7bd4001750 con 0x7f7be409f8f0 2024-05-21T17:23:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: dump: 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.123 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 5 monclient: authenticate success, global_id 4542 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7bd40018f0 con 0x7f7be409f8f0 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 1 -- 172.21.15.3:0/4146436531 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7bd4003620 con 0x7f7be409f8f0 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 4 set_mon_vals no callback set 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be886a640 10 monclient: dump: 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.127 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.127 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient: shutdown 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- 172.21.15.3:0/4146436531 >> v1:172.21.15.3:6792/0 conn(0x7f7be409f8f0 legacy=0x7f7be4141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- 172.21.15.3:0/4146436531 shutdown_connections 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- 172.21.15.3:0/4146436531 >> 172.21.15.3:0/4146436531 conn(0x7f7be40a72c0 msgr2=0x7f7be40a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- 172.21.15.3:0/4146436531 shutdown_connections 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- 172.21.15.3:0/4146436531 wait complete. 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient: build_initial_monmap 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient: monmap: 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:44.129 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 Processor -- start 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- start start 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient: init 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient: _add_conns ranks=[5,6,7,0,3,4,8,2,1] 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient(hunting): picked mon.noname-h con 0x7f7be409f8f0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient(hunting): picked mon.noname-d con 0x7f7be413d0e0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient(hunting): picked mon.noname-i con 0x7f7be41436d0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be4070690 con 0x7f7be409f8f0 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be40a30c0 con 0x7f7be413d0e0 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be413c9f0 con 0x7f7be41436d0 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:44.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7bebad0640 10 monclient(hunting): authenticate will time out at 1091.275314s 2024-05-21T17:23:44.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be906b640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f7be413d0e0 0x7f7be41d6e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:38528/0 (socket says 172.21.15.3:38528) 2024-05-21T17:23:44.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.112+0000 7f7be906b640 1 -- 172.21.15.3:0/1408432181 learned_addr learned my addr 172.21.15.3:0/1408432181 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:44.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2460911527 0 0) 0x7f7be40a30c0 con 0x7f7be413d0e0 2024-05-21T17:23:44.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 10 monclient(hunting): my global_id is 4551 2024-05-21T17:23:44.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bc8003530 con 0x7f7be413d0e0 2024-05-21T17:23:44.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1459607659 0 0) 0x7f7bc8003530 con 0x7f7be413d0e0 2024-05-21T17:23:44.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7be40a30c0 con 0x7f7be413d0e0 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7bd4001710 con 0x7f7be413d0e0 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 10 monclient(hunting): dump: 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.136 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.116+0000 7f7bda7fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1674764164 0 0) 0x7f7be40a30c0 con 0x7f7be413d0e0 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 >> v1:172.21.15.203:6792/0 conn(0x7f7be41436d0 legacy=0x7f7be41da590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 >> v1:172.21.15.203:6791/0 conn(0x7f7be409f8f0 legacy=0x7f7be4072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 monclient: found mon.d 2024-05-21T17:23:44.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be4071960 con 0x7f7be413d0e0 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 5 monclient: authenticate success, global_id 4551 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 10 monclient: _renew_subs 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7be40710c0 con 0x7f7be413d0e0 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 10 monclient: _renew_subs 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7bd4003b90 con 0x7f7be413d0e0 2024-05-21T17:23:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f7be40735a0 con 0x7f7be413d0e0 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7bd4004180 con 0x7f7be413d0e0 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 10 monclient: dump: 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.139 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bdbfff640 4 set_mon_vals no callback set 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f7bd40043a0 con 0x7f7be413d0e0 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 --> v1:172.21.15.3:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bcc005090 con 0x7f7be413d0e0 2024-05-21T17:23:44.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f7bd4045d40 con 0x7f7be413d0e0 2024-05-21T17:23:44.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.120+0000 7f7be986c640 10 monclient: get_auth_request con 0x7f7bc8038c60 auth_method 0 2024-05-21T17:23:44.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.124+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mon.6 v1:172.21.15.3:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f7bd4019a50 con 0x7f7be413d0e0 2024-05-21T17:23:44.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.124+0000 7f7bda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:44.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.124+0000 7f7bda7fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:44.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.284+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f7bcc000ba0 con 0x7f7bc8038c60 2024-05-21T17:23:44.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.284+0000 7f7bda7fc640 1 -- 172.21.15.3:0/1408432181 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7f7bcc000ba0 con 0x7f7bc8038c60 2024-05-21T17:23:44.289 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 2 to deep-scrub 2024-05-21T17:23:44.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.288+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 >> v1:172.21.15.203:6812/343827626 conn(0x7f7bc8038c60 legacy=0x7f7bc803b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.288+0000 7f7bebad0640 10 monclient: shutdown 2024-05-21T17:23:44.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.288+0000 7f7bebad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:44.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.288+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 >> v1:172.21.15.3:6792/0 conn(0x7f7be413d0e0 legacy=0x7f7be41d6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.288+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 shutdown_connections 2024-05-21T17:23:44.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.288+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 >> 172.21.15.3:0/1408432181 conn(0x7f7be40a72c0 msgr2=0x7f7be40a5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:44.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.288+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 shutdown_connections 2024-05-21T17:23:44.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.288+0000 7f7bebad0640 1 -- 172.21.15.3:0/1408432181 wait complete. 2024-05-21T17:23:44.306 INFO:tasks.ceph:Scrubbing osd.3 2024-05-21T17:23:44.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient: get_monmap_and_config 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient: build_initial_monmap 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient: monmap: 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:44.433406+0000 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:44.433406+0000 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:44.433 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:44.433 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:44.433 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:44.433 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:44.433 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:44.433 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:44.433 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:44.433 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 1 Processor -- start 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 1 -- start start 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient: init 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient: _add_conns ranks=[5,0,2,8,3,7,1,6,4] 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient(hunting): picked mon.noname-h con 0x7f4358135bb0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient(hunting): picked mon.noname-a con 0x7f43580a3df0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient(hunting): picked mon.noname-b con 0x7f435813f470 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43580a7920 con 0x7f43580a3df0 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43580a2b20 con 0x7f435813f470 2024-05-21T17:23:44.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43580a2e00 con 0x7f4358135bb0 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435f913640 10 monclient(hunting): authenticate will time out at 1091.591022s 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435deb0640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f435813f470 0x7f4358141860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:41092/0 (socket says 172.21.15.3:41092) 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435deb0640 1 -- 172.21.15.3:0/4001006725 learned_addr learned my addr 172.21.15.3:0/4001006725 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f435ceae640 1 --1- 172.21.15.3:0/4001006725 >> v1:172.21.15.3:6789/0 conn(0x7f43580a3df0 0x7f435813dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38558/0 (socket says 172.21.15.3:38558) 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484439749 0 0) 0x7f43580a2b20 con 0x7f435813f470 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient(hunting): my global_id is 134282 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4344003530 con 0x7f435813f470 2024-05-21T17:23:44.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225035273 0 0) 0x7f43580a7920 con 0x7f43580a3df0 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient(hunting): my global_id is 4761 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43580a2b20 con 0x7f43580a3df0 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3757731300 0 0) 0x7f4344003530 con 0x7f435813f470 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 >> v1:172.21.15.203:6791/0 conn(0x7f4358135bb0 legacy=0x7f4358135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 >> v1:172.21.15.3:6789/0 conn(0x7f43580a3df0 legacy=0x7f435813dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 monclient: found mon.noname-b 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:44.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43580a7290 con 0x7f435813f470 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient: _finish_auth 0 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4350001bc0 con 0x7f435813f470 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.428+0000 7f434ffff640 10 monclient: dump: 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.444 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 5 monclient: authenticate success, global_id 134282 2024-05-21T17:23:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4350001d60 con 0x7f435813f470 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434ffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 4 set_mon_vals no callback set 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434ffff640 1 -- 172.21.15.3:0/4001006725 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4350004f70 con 0x7f435813f470 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434ffff640 10 monclient: dump: 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: shutdown 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- 172.21.15.3:0/4001006725 >> v1:172.21.15.3:6790/0 conn(0x7f435813f470 legacy=0x7f4358141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- 172.21.15.3:0/4001006725 shutdown_connections 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- 172.21.15.3:0/4001006725 >> 172.21.15.3:0/4001006725 conn(0x7f43580ab8d0 msgr2=0x7f43580a7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- 172.21.15.3:0/4001006725 shutdown_connections 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- 172.21.15.3:0/4001006725 wait complete. 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: build_initial_monmap 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: monmap: 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:44.448 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 Processor -- start 2024-05-21T17:23:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- start start 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: init 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: _add_conns ranks=[2,6,7,5,1,4,0,8,3] 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient(hunting): picked mon.noname-b con 0x7f43580a3df0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient(hunting): picked mon.noname-d con 0x7f4358135bb0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient(hunting): picked mon.noname-i con 0x7f435813f470 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f435813e5d0 con 0x7f43580a3df0 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f435813b990 con 0x7f4358135bb0 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43580a75f0 con 0x7f435813f470 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient(hunting): authenticate will time out at 1091.594279s 2024-05-21T17:23:44.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435d6af640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f43580a3df0 0x7f435813ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:41104/0 (socket says 172.21.15.3:41104) 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435d6af640 1 -- 172.21.15.3:0/2655280126 learned_addr learned my addr 172.21.15.3:0/2655280126 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435ceae640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7f4358135bb0 0x7f435813b280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:38530/0 (socket says 172.21.15.3:38530) 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1480555569 0 0) 0x7f435813e5d0 con 0x7f43580a3df0 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): my global_id is 134291 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f433c003530 con 0x7f43580a3df0 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 559760270 0 0) 0x7f435813b990 con 0x7f4358135bb0 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): my global_id is 4560 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f435813e5d0 con 0x7f4358135bb0 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1936062228 0 0) 0x7f435813e5d0 con 0x7f4358135bb0 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f435813b990 con 0x7f4358135bb0 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4348001a60 con 0x7f4358135bb0 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): dump: 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.452 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2910520905 0 0) 0x7f433c003530 con 0x7f43580a3df0 2024-05-21T17:23:44.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f435813e5d0 con 0x7f43580a3df0 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4354001b40 con 0x7f43580a3df0 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): dump: 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.454 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3427938984 0 0) 0x7f43580a75f0 con 0x7f435813f470 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient(hunting): my global_id is 4786 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f433c003530 con 0x7f435813f470 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 707019661 0 0) 0x7f435813b990 con 0x7f4358135bb0 2024-05-21T17:23:44.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 >> v1:172.21.15.203:6792/0 conn(0x7f435813f470 legacy=0x7f43581d6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 >> v1:172.21.15.3:6790/0 conn(0x7f43580a3df0 legacy=0x7f435813ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient: _finish_hunting 0 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 monclient: found mon.d 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435809cc80 con 0x7f4358135bb0 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient: _finish_auth 0 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient: _check_auth_tickets 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 5 monclient: authenticate success, global_id 4560 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: _renew_subs 2024-05-21T17:23:44.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f435813c5c0 con 0x7f4358135bb0 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: _renew_subs 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f435813dce0 con 0x7f4358135bb0 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4348001cc0 con 0x7f4358135bb0 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4348004de0 con 0x7f4358135bb0 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434f7fe640 4 set_mon_vals no callback set 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.432+0000 7f434dffb640 10 monclient: dump: 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.458 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f435f913640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.3:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4338000f90 con 0x7f4358135bb0 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f4348004f80 con 0x7f4358135bb0 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f43480459d0 con 0x7f4358135bb0 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.203:6804/1468009857 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f433c043cd0 con 0x7f433c040060 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== mon.6 v1:172.21.15.3:6792/0 9 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (unknown 3512546819 0 0) 0x7f4348045de0 con 0x7f4358135bb0 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f434dffb640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f435d6af640 10 monclient: get_auth_request con 0x7f433c0394b0 auth_method 0 2024-05-21T17:23:44.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f435deb0640 10 monclient: get_auth_request con 0x7f433c040060 auth_method 0 2024-05-21T17:23:44.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.436+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== osd.3 v1:172.21.15.203:6804/1468009857 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f433c043cd0 con 0x7f433c040060 2024-05-21T17:23:44.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 --> v1:172.21.15.203:6804/1468009857 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4338002ce0 con 0x7f433c040060 2024-05-21T17:23:44.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f434dffb640 1 -- 172.21.15.3:0/2655280126 <== osd.3 v1:172.21.15.203:6804/1468009857 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f4338002ce0 con 0x7f433c040060 2024-05-21T17:23:44.460 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-21T17:23:44.461 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:44.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 >> v1:172.21.15.203:6804/1468009857 conn(0x7f433c040060 legacy=0x7f433c042480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 >> v1:172.21.15.203:6812/343827626 conn(0x7f433c0394b0 legacy=0x7f433c03b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 10 monclient: shutdown 2024-05-21T17:23:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 >> v1:172.21.15.3:6792/0 conn(0x7f4358135bb0 legacy=0x7f435813b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435deb0640 1 -- 172.21.15.3:0/2655280126 reap_dead start 2024-05-21T17:23:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 shutdown_connections 2024-05-21T17:23:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 >> 172.21.15.3:0/2655280126 conn(0x7f43580ab8d0 msgr2=0x7f4358073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 shutdown_connections 2024-05-21T17:23:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.452+0000 7f435f913640 1 -- 172.21.15.3:0/2655280126 wait complete. 2024-05-21T17:23:44.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-21T17:23:44.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient: get_monmap_and_config 2024-05-21T17:23:44.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient: build_initial_monmap 2024-05-21T17:23:44.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient: monmap: 2024-05-21T17:23:44.587 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:44.587 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.587 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:44.588715+0000 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:44.588715+0000 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:44.588 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:44.589 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 Processor -- start 2024-05-21T17:23:44.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- start start 2024-05-21T17:23:44.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient: init 2024-05-21T17:23:44.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient: _add_conns ranks=[8,5,6,4,0,1,3,2,7] 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient(hunting): picked mon.noname-e con 0x7f79680a4ac0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient(hunting): picked mon.noname-h con 0x7f79680a11a0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient(hunting): picked mon.noname-d con 0x7f79680a55b0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796806e050 con 0x7f79680a11a0 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796806c920 con 0x7f79680a55b0 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79680a6580 con 0x7f79680a4ac0 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient(hunting): authenticate will time out at 1091.746276s 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796e51c640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f79680a4ac0 0x7f79680a4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:55086/0 (socket says 172.21.15.3:55086) 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796e51c640 1 -- 172.21.15.3:0/1044022174 learned_addr learned my addr 172.21.15.3:0/1044022174 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1838120389 0 0) 0x7f79680a6580 con 0x7f79680a4ac0 2024-05-21T17:23:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient(hunting): my global_id is 4778 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7954003530 con 0x7f79680a4ac0 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845695677 0 0) 0x7f796806c920 con 0x7f79680a55b0 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient(hunting): my global_id is 4569 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79680a6580 con 0x7f79680a55b0 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2534503957 0 0) 0x7f79680a6580 con 0x7f79680a55b0 2024-05-21T17:23:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 >> v1:172.21.15.3:6793/0 conn(0x7f79680a4ac0 legacy=0x7f79680a4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 >> v1:172.21.15.203:6791/0 conn(0x7f79680a11a0 legacy=0x7f796813dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: _finish_hunting 0 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 monclient: found mon.noname-d 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796809ff30 con 0x7f79680a55b0 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: _finish_auth 0 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: _check_auth_tickets 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7960001ba0 con 0x7f79680a55b0 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 5 monclient: authenticate success, global_id 4569 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:44.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: dump: 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.599 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 <== mon.6 v1:172.21.15.3:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7960001dc0 con 0x7f79680a55b0 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 1 -- 172.21.15.3:0/1044022174 <== mon.6 v1:172.21.15.3:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7960003ce0 con 0x7f79680a55b0 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 4 set_mon_vals no callback set 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:44.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f796d51a640 10 monclient: dump: 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.601 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 10 monclient: shutdown 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- 172.21.15.3:0/1044022174 >> v1:172.21.15.3:6792/0 conn(0x7f79680a55b0 legacy=0x7f79681415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- 172.21.15.3:0/1044022174 shutdown_connections 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- 172.21.15.3:0/1044022174 >> 172.21.15.3:0/1044022174 conn(0x7f79680a96b0 msgr2=0x7f79680a5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- 172.21.15.3:0/1044022174 shutdown_connections 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.584+0000 7f7970780640 1 -- 172.21.15.3:0/1044022174 wait complete. 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: build_initial_monmap 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: monmap: 2024-05-21T17:23:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 1 Processor -- start 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 1 -- start start 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: init 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: _add_conns ranks=[3,2,5,6,7,1,4,0,8] 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient(hunting): picked mon.noname-g con 0x7f79680a11a0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient(hunting): picked mon.noname-b con 0x7f79680a4ac0 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient(hunting): picked mon.noname-h con 0x7f79680a55b0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796813e6f0 con 0x7f79680a4ac0 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79681cd990 con 0x7f79680a11a0 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796813e4b0 con 0x7f79680a55b0 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:44.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient(hunting): authenticate will time out at 1091.749404s 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f796dd1b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f79680a4ac0 0x7f79681d2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:41114/0 (socket says 172.21.15.3:41114) 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f796dd1b640 1 -- 172.21.15.3:0/1401916847 learned_addr learned my addr 172.21.15.3:0/1401916847 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335760518 0 0) 0x7f796813e6f0 con 0x7f79680a4ac0 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): my global_id is 134300 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7948003530 con 0x7f79680a4ac0 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1418066531 0 0) 0x7f7948003530 con 0x7f79680a4ac0 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f796813e6f0 con 0x7f79680a4ac0 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7958001be0 con 0x7f79680a4ac0 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:44.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): dump: 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.606 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1062220674 0 0) 0x7f796813e4b0 con 0x7f79680a55b0 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): my global_id is 4802 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7948003530 con 0x7f79680a55b0 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3336346259 0 0) 0x7f79681cd990 con 0x7f79680a11a0 2024-05-21T17:23:44.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient(hunting): my global_id is 4719 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f796813e4b0 con 0x7f79680a11a0 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2639095008 0 0) 0x7f796813e6f0 con 0x7f79680a4ac0 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 >> v1:172.21.15.203:6791/0 conn(0x7f79680a55b0 legacy=0x7f79681d5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 >> v1:172.21.15.203:6790/0 conn(0x7f79680a11a0 legacy=0x7f7968142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient: _finish_hunting 0 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 monclient: found mon.b 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796809f430 con 0x7f79680a4ac0 2024-05-21T17:23:44.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient: _finish_auth 0 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient: _check_auth_tickets 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 5 monclient: authenticate success, global_id 134300 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: _renew_subs 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f796813fc40 con 0x7f79680a4ac0 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: _renew_subs 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7968142890 con 0x7f79680a4ac0 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79580038d0 con 0x7f79680a4ac0 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f7958003ec0 con 0x7f79680a4ac0 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 10 monclient: dump: 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.610 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7967fff640 4 set_mon_vals no callback set 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f79580040e0 con 0x7f79680a4ac0 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f794c005090 con 0x7f79680a4ac0 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f7958045d40 con 0x7f79680a4ac0 2024-05-21T17:23:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.588+0000 7f796e51c640 10 monclient: get_auth_request con 0x7f7948038ea0 auth_method 0 2024-05-21T17:23:44.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.596+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f796806e050 con 0x7f79680a4ac0 2024-05-21T17:23:44.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.596+0000 7f7966ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:44.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.596+0000 7f7966ffd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:44.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.752+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f794c000ba0 con 0x7f7948038ea0 2024-05-21T17:23:44.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.752+0000 7f7966ffd640 1 -- 172.21.15.3:0/1401916847 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7f794c000ba0 con 0x7f7948038ea0 2024-05-21T17:23:44.757 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 3 to deep-scrub 2024-05-21T17:23:44.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.756+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 >> v1:172.21.15.203:6812/343827626 conn(0x7f7948038ea0 legacy=0x7f794803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.756+0000 7f7970780640 10 monclient: shutdown 2024-05-21T17:23:44.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.756+0000 7f7970780640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:44.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.756+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 >> v1:172.21.15.3:6790/0 conn(0x7f79680a4ac0 legacy=0x7f79681d2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.756+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 shutdown_connections 2024-05-21T17:23:44.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.756+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 >> 172.21.15.3:0/1401916847 conn(0x7f79680a96b0 msgr2=0x7f79680a7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:44.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.756+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 shutdown_connections 2024-05-21T17:23:44.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.756+0000 7f7970780640 1 -- 172.21.15.3:0/1401916847 wait complete. 2024-05-21T17:23:44.774 INFO:tasks.ceph:Scrubbing osd.4 2024-05-21T17:23:44.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-21T17:23:44.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient: get_monmap_and_config 2024-05-21T17:23:44.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient: build_initial_monmap 2024-05-21T17:23:44.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient: monmap: 2024-05-21T17:23:44.903 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:44.903 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.903 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:44.904711+0000 2024-05-21T17:23:44.903 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:44.904711+0000 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:44.904 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 Processor -- start 2024-05-21T17:23:44.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- start start 2024-05-21T17:23:44.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient: init 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient: _add_conns ranks=[1,8,2,6,5,7,3,0,4] 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient(hunting): picked mon.noname-f con 0x7fde0c0a5bd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient(hunting): picked mon.noname-e con 0x7fde0c0a66c0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient(hunting): picked mon.noname-b con 0x7fde0c09db30 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c09d430 con 0x7fde0c0a5bd0 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0a7680 con 0x7fde0c09db30 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0a6d00 con 0x7fde0c0a66c0 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient(hunting): authenticate will time out at 1092.062136s 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0bfff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fde0c09db30 0x7fde0c1415f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:41122/0 (socket says 172.21.15.3:41122) 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0bfff640 1 -- 172.21.15.3:0/2942522270 learned_addr learned my addr 172.21.15.3:0/2942522270 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:44.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2934963212 0 0) 0x7fde0c0a7680 con 0x7fde0c09db30 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient(hunting): my global_id is 134309 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf8003530 con 0x7fde0c09db30 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967776067 0 0) 0x7fde0c0a6d00 con 0x7fde0c0a66c0 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient(hunting): my global_id is 4787 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde0c0a7680 con 0x7fde0c0a66c0 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3364079044 0 0) 0x7fddf8003530 con 0x7fde0c09db30 2024-05-21T17:23:44.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 >> v1:172.21.15.3:6793/0 conn(0x7fde0c0a66c0 legacy=0x7fde0c13dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 >> v1:172.21.15.203:6789/0 conn(0x7fde0c0a5bd0 legacy=0x7fde0c0a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 monclient: found mon.noname-b 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c09df40 con 0x7fde0c09db30 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fde04001840 con 0x7fde0c09db30 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 5 monclient: authenticate success, global_id 134309 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:44.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: dump: 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.914 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde04001a60 con 0x7fde0c09db30 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 1 -- 172.21.15.3:0/2942522270 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fde04004bf0 con 0x7fde0c09db30 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde0a7fc640 10 monclient: dump: 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 4 set_mon_vals no callback set 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 10 monclient: shutdown 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- 172.21.15.3:0/2942522270 >> v1:172.21.15.3:6790/0 conn(0x7fde0c09db30 legacy=0x7fde0c1415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- 172.21.15.3:0/2942522270 shutdown_connections 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- 172.21.15.3:0/2942522270 >> 172.21.15.3:0/2942522270 conn(0x7fde0c0a7ea0 msgr2=0x7fde0c0a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- 172.21.15.3:0/2942522270 shutdown_connections 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.900+0000 7fde11aef640 1 -- 172.21.15.3:0/2942522270 wait complete. 2024-05-21T17:23:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: build_initial_monmap 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: monmap: 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 1 Processor -- start 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 1 -- start start 2024-05-21T17:23:44.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: init 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: _add_conns ranks=[7,3,6,2,4,8,5,1,0] 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient(hunting): picked mon.noname-i con 0x7fde0c09db30 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient(hunting): picked mon.noname-g con 0x7fde0c0a5bd0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient(hunting): picked mon.noname-d con 0x7fde0c0a66c0 addr v1:172.21.15.3:6792/0 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c09e730 con 0x7fde0c0a5bd0 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c1ce3f0 con 0x7fde0c0a66c0 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c138850 con 0x7fde0c09db30 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient(hunting): authenticate will time out at 1092.065493s 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde0bfff640 1 --1- >> v1:172.21.15.3:6792/0 conn(0x7fde0c0a66c0 0x7fde0c1d5910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6792/0 says I am v1:172.21.15.3:38550/0 (socket says 172.21.15.3:38550) 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde0bfff640 1 -- 172.21.15.3:0/2605720996 learned_addr learned my addr 172.21.15.3:0/2605720996 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2855188876 0 0) 0x7fde0c1ce3f0 con 0x7fde0c0a66c0 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): my global_id is 4578 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddec003530 con 0x7fde0c0a66c0 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1069890047 0 0) 0x7fddec003530 con 0x7fde0c0a66c0 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde0c1ce3f0 con 0x7fde0c0a66c0 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fde04004b30 con 0x7fde0c0a66c0 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): dump: 2024-05-21T17:23:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.3 v1:172.21.15.203:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763729670 0 0) 0x7fddf40010b0 con 0x7fde0c0a5bd0 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): my global_id is 4728 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.203:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddec003530 con 0x7fde0c0a5bd0 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 311542415 0 0) 0x7fde0c138850 con 0x7fde0c09db30 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient(hunting): my global_id is 4795 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf40010b0 con 0x7fde0c09db30 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2101592489 0 0) 0x7fde0c1ce3f0 con 0x7fde0c0a66c0 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 >> v1:172.21.15.203:6792/0 conn(0x7fde0c09db30 legacy=0x7fde0c1cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 >> v1:172.21.15.203:6790/0 conn(0x7fde0c0a5bd0 legacy=0x7fde0c1cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 monclient: found mon.d 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.3:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c13a8e0 con 0x7fde0c0a66c0 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: _finish_auth 0 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 5 monclient: authenticate success, global_id 4578 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: _renew_subs 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.3:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fde0c13a320 con 0x7fde0c0a66c0 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: _renew_subs 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.3:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fde0c139580 con 0x7fde0c0a66c0 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde04001140 con 0x7fde0c0a66c0 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fde04001960 con 0x7fde0c0a66c0 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde09ffb640 4 set_mon_vals no callback set 2024-05-21T17:23:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: dump: 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.3:6792/0 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.3:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fddf0000f90 con 0x7fde0c0a66c0 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fde04010210 con 0x7fde0c0a66c0 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fde04045d10 con 0x7fde0c0a66c0 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.203:6800/4020495055 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fddec0436e0 con 0x7fddec03fa70 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== mon.6 v1:172.21.15.3:6792/0 9 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (unknown 3512546819 0 0) 0x7fde040460e0 con 0x7fde0c0a66c0 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.904+0000 7fde03fff640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.908+0000 7fde0b7fe640 10 monclient: get_auth_request con 0x7fddec038eb0 auth_method 0 2024-05-21T17:23:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.908+0000 7fde0affd640 10 monclient: get_auth_request con 0x7fddec03fa70 auth_method 0 2024-05-21T17:23:44.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.908+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== osd.4 v1:172.21.15.203:6800/4020495055 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fddec0436e0 con 0x7fddec03fa70 2024-05-21T17:23:44.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.920+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 --> v1:172.21.15.203:6800/4020495055 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fddf0002ce0 con 0x7fddec03fa70 2024-05-21T17:23:44.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde03fff640 1 -- 172.21.15.3:0/2605720996 <== osd.4 v1:172.21.15.203:6800/4020495055 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fddf0002ce0 con 0x7fddec03fa70 2024-05-21T17:23:44.928 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:44.929 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-21T17:23:44.929 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:44.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 >> v1:172.21.15.203:6800/4020495055 conn(0x7fddec03fa70 legacy=0x7fddec041e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 >> v1:172.21.15.203:6812/343827626 conn(0x7fddec038eb0 legacy=0x7fddec03b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 10 monclient: shutdown 2024-05-21T17:23:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 >> v1:172.21.15.3:6792/0 conn(0x7fde0c0a66c0 legacy=0x7fde0c1d5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde0bfff640 1 -- 172.21.15.3:0/2605720996 reap_dead start 2024-05-21T17:23:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 shutdown_connections 2024-05-21T17:23:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 >> 172.21.15.3:0/2605720996 conn(0x7fde0c0a7ea0 msgr2=0x7fde0c0a38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 shutdown_connections 2024-05-21T17:23:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:44.924+0000 7fde11aef640 1 -- 172.21.15.3:0/2605720996 wait complete. 2024-05-21T17:23:44.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-21T17:23:45.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient: get_monmap_and_config 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient: build_initial_monmap 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient: monmap: 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:45.060255+0000 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:45.060255+0000 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:45.060 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 Processor -- start 2024-05-21T17:23:45.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- start start 2024-05-21T17:23:45.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient: init 2024-05-21T17:23:45.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:45.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient: _add_conns ranks=[7,4,1,6,3,5,8,0,2] 2024-05-21T17:23:45.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient(hunting): picked mon.noname-i con 0x7fac080a5bd0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:45.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient(hunting): picked mon.noname-c con 0x7fac080a66c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:45.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient(hunting): picked mon.noname-f con 0x7fac080a70c0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac0809e480 con 0x7fac080a70c0 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac0809cce0 con 0x7fac080a66c0 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac0809cfc0 con 0x7fac080a5bd0 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient(hunting): authenticate will time out at 1092.217815s 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0cfe3640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fac080a66c0 0x7fac0813dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:54980/0 (socket says 172.21.15.3:54980) 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0cfe3640 1 -- 172.21.15.3:0/464279955 learned_addr learned my addr 172.21.15.3:0/464279955 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205496863 0 0) 0x7fac0809cce0 con 0x7fac080a66c0 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient(hunting): my global_id is 4720 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabf4003530 con 0x7fac080a66c0 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3883551929 0 0) 0x7fabf4003530 con 0x7fac080a66c0 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 >> v1:172.21.15.203:6792/0 conn(0x7fac080a5bd0 legacy=0x7fac080a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 >> v1:172.21.15.203:6789/0 conn(0x7fac080a70c0 legacy=0x7fac08141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 monclient: found mon.noname-c 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac080742e0 con 0x7fac080a66c0 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: _finish_auth 0 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:45.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fac04001bc0 con 0x7fac080a66c0 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 5 monclient: authenticate success, global_id 4720 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: dump: 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac04001de0 con 0x7fac080a66c0 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 1 -- 172.21.15.3:0/464279955 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fac04004f70 con 0x7fac080a66c0 2024-05-21T17:23:45.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 4 set_mon_vals no callback set 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fabfffff640 10 monclient: dump: 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 10 monclient: shutdown 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- 172.21.15.3:0/464279955 >> v1:172.21.15.3:6791/0 conn(0x7fac080a66c0 legacy=0x7fac0813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- 172.21.15.3:0/464279955 shutdown_connections 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- 172.21.15.3:0/464279955 >> 172.21.15.3:0/464279955 conn(0x7fac080ab7d0 msgr2=0x7fac080a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- 172.21.15.3:0/464279955 shutdown_connections 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.056+0000 7fac0fa48640 1 -- 172.21.15.3:0/464279955 wait complete. 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: build_initial_monmap 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: monmap: 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:45.074 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 1 Processor -- start 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 1 -- start start 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: init 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: _add_conns ranks=[8,4,5,3,1,2,0,6,7] 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient(hunting): picked mon.noname-e con 0x7fac080a5bd0 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:45.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient(hunting): picked mon.noname-c con 0x7fac080a66c0 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient(hunting): picked mon.noname-h con 0x7fac080a70c0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac0813f9a0 con 0x7fac080a66c0 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac08141c90 con 0x7fac080a70c0 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac081413f0 con 0x7fac080a5bd0 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient(hunting): authenticate will time out at 1092.221137s 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0cfe3640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7fac080a66c0 0x7fac081cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:54984/0 (socket says 172.21.15.3:54984) 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0cfe3640 1 -- 172.21.15.3:0/972463479 learned_addr learned my addr 172.21.15.3:0/972463479 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200228182 0 0) 0x7fac0813f9a0 con 0x7fac080a66c0 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): my global_id is 4729 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabec003530 con 0x7fac080a66c0 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3969641639 0 0) 0x7fac081413f0 con 0x7fac080a5bd0 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): my global_id is 4796 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac0813f9a0 con 0x7fac080a5bd0 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3195709056 0 0) 0x7fabec003530 con 0x7fac080a66c0 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac081413f0 con 0x7fac080a66c0 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fac04004eb0 con 0x7fac080a66c0 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): dump: 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 866026850 0 0) 0x7fac0813f9a0 con 0x7fac080a5bd0 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabec003530 con 0x7fac080a5bd0 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fabf80019d0 con 0x7fac080a5bd0 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): dump: 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.080 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 502107998 0 0) 0x7fac08141c90 con 0x7fac080a70c0 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient(hunting): my global_id is 4811 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac0813f9a0 con 0x7fac080a70c0 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1153310107 0 0) 0x7fac081413f0 con 0x7fac080a66c0 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 >> v1:172.21.15.3:6793/0 conn(0x7fac080a5bd0 legacy=0x7fac081420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 >> v1:172.21.15.203:6791/0 conn(0x7fac080a70c0 legacy=0x7fac081d5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient: _finish_hunting 0 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 monclient: found mon.c 2024-05-21T17:23:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac081427c0 con 0x7fac080a66c0 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient: _finish_auth 0 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient: _check_auth_tickets 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 5 monclient: authenticate success, global_id 4729 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: _renew_subs 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fac081cdef0 con 0x7fac080a66c0 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: _renew_subs 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fac081d6ff0 con 0x7fac080a66c0 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac04005270 con 0x7fac080a66c0 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fac04005760 con 0x7fac080a66c0 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabff7fe640 4 set_mon_vals no callback set 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 10 monclient: dump: 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fac040108e0 con 0x7fac080a66c0 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac080a79d0 con 0x7fac080a66c0 2024-05-21T17:23:45.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fac04045f40 con 0x7fac080a66c0 2024-05-21T17:23:45.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.060+0000 7fac0d7e4640 10 monclient: get_auth_request con 0x7fabec039390 auth_method 0 2024-05-21T17:23:45.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.068+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7fac04016180 con 0x7fac080a66c0 2024-05-21T17:23:45.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.068+0000 7fabfdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.068+0000 7fabfdffb640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:45.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.240+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fac081cd400 con 0x7fabec039390 2024-05-21T17:23:45.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.240+0000 7fabfdffb640 1 -- 172.21.15.3:0/972463479 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7fac081cd400 con 0x7fabec039390 2024-05-21T17:23:45.245 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 4 to deep-scrub 2024-05-21T17:23:45.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.244+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 >> v1:172.21.15.203:6812/343827626 conn(0x7fabec039390 legacy=0x7fabec03b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.244+0000 7fac0fa48640 10 monclient: shutdown 2024-05-21T17:23:45.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.244+0000 7fac0fa48640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:45.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.244+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 >> v1:172.21.15.3:6791/0 conn(0x7fac080a66c0 legacy=0x7fac081cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.244+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 shutdown_connections 2024-05-21T17:23:45.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.244+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 >> 172.21.15.3:0/972463479 conn(0x7fac080ab7d0 msgr2=0x7fac080a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:45.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.244+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 shutdown_connections 2024-05-21T17:23:45.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.244+0000 7fac0fa48640 1 -- 172.21.15.3:0/972463479 wait complete. 2024-05-21T17:23:45.263 INFO:tasks.ceph:Scrubbing osd.5 2024-05-21T17:23:45.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-21T17:23:45.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient: get_monmap_and_config 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient: build_initial_monmap 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient: monmap: 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:45.385479+0000 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:45.385479+0000 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:45.385 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 1 Processor -- start 2024-05-21T17:23:45.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 1 -- start start 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient: init 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient: _add_conns ranks=[5,0,4,6,2,1,3,7,8] 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient(hunting): picked mon.noname-h con 0x7f2a600a5bd0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient(hunting): picked mon.noname-a con 0x7f2a600a66c0 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient(hunting): picked mon.noname-c con 0x7f2a6009db30 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6009d430 con 0x7f2a600a66c0 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a600a7680 con 0x7f2a6009db30 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a600a6d00 con 0x7f2a600a5bd0 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a67875640 10 monclient(hunting): authenticate will time out at 1092.543469s 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a65e12640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f2a6009db30 0x7f2a601415f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:54990/0 (socket says 172.21.15.3:54990) 2024-05-21T17:23:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a65e12640 1 -- 172.21.15.3:0/118979226 learned_addr learned my addr 172.21.15.3:0/118979226 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1684804886 0 0) 0x7f2a600a7680 con 0x7f2a6009db30 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a57fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a57fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a57fff640 10 monclient(hunting): my global_id is 4738 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.380+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a4c003530 con 0x7f2a6009db30 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292140228 0 0) 0x7f2a6009d430 con 0x7f2a600a66c0 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient(hunting): my global_id is 4770 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a600a7680 con 0x7f2a600a66c0 2024-05-21T17:23:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1875997838 0 0) 0x7f2a4c003530 con 0x7f2a6009db30 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 >> v1:172.21.15.203:6791/0 conn(0x7f2a600a5bd0 legacy=0x7f2a600a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 >> v1:172.21.15.3:6789/0 conn(0x7f2a600a66c0 legacy=0x7f2a6013dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: _finish_hunting 0 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 monclient: found mon.noname-c 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6009df40 con 0x7f2a6009db30 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: _finish_auth 0 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: _check_auth_tickets 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2a58001bc0 con 0x7f2a6009db30 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: dump: 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.395 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 5 monclient: authenticate success, global_id 4738 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 <== mon.4 v1:172.21.15.3:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a58001de0 con 0x7f2a6009db30 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 1 -- 172.21.15.3:0/118979226 <== mon.4 v1:172.21.15.3:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2a58004f70 con 0x7f2a6009db30 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 4 set_mon_vals no callback set 2024-05-21T17:23:45.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a57fff640 10 monclient: dump: 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.397 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: shutdown 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- 172.21.15.3:0/118979226 >> v1:172.21.15.3:6791/0 conn(0x7f2a6009db30 legacy=0x7f2a601415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- 172.21.15.3:0/118979226 shutdown_connections 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- 172.21.15.3:0/118979226 >> 172.21.15.3:0/118979226 conn(0x7f2a600a7ea0 msgr2=0x7f2a600a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- 172.21.15.3:0/118979226 shutdown_connections 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- 172.21.15.3:0/118979226 wait complete. 2024-05-21T17:23:45.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: build_initial_monmap 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: monmap: 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 Processor -- start 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- start start 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: init 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: _add_conns ranks=[2,1,3,7,8,6,0,5,4] 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient(hunting): picked mon.noname-b con 0x7f2a6009db30 addr v1:172.21.15.3:6790/0 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient(hunting): picked mon.noname-f con 0x7f2a600a5bd0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient(hunting): picked mon.noname-g con 0x7f2a600a66c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6006c050 con 0x7f2a600a5bd0 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a60138fb0 con 0x7f2a6009db30 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6009e730 con 0x7f2a600a66c0 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient(hunting): authenticate will time out at 1092.546507s 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a65611640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f2a6009db30 0x7f2a60135350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:41124/0 (socket says 172.21.15.3:41124) 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a65611640 1 -- 172.21.15.3:0/3378800290 learned_addr learned my addr 172.21.15.3:0/3378800290 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2772752391 0 0) 0x7f2a60138fb0 con 0x7f2a6009db30 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient(hunting): my global_id is 134318 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a44003530 con 0x7f2a6009db30 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 955283472 0 0) 0x7f2a44003530 con 0x7f2a6009db30 2024-05-21T17:23:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a60138fb0 con 0x7f2a6009db30 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2a5c001ba0 con 0x7f2a6009db30 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient(hunting): dump: 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.402 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2873858777 0 0) 0x7f2a60138fb0 con 0x7f2a6009db30 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 >> v1:172.21.15.203:6790/0 conn(0x7f2a600a66c0 legacy=0x7f2a601d5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 >> v1:172.21.15.203:6789/0 conn(0x7f2a600a5bd0 legacy=0x7f2a60135a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient: _finish_hunting 0 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 monclient: found mon.b 2024-05-21T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a600a78d0 con 0x7f2a6009db30 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient: _finish_auth 0 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient: _check_auth_tickets 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 5 monclient: authenticate success, global_id 134318 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: _renew_subs 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a60138cd0 con 0x7f2a6009db30 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: _renew_subs 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a60137820 con 0x7f2a6009db30 2024-05-21T17:23:45.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a5c0038d0 con 0x7f2a6009db30 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f2a5c003ec0 con 0x7f2a6009db30 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a577fe640 4 set_mon_vals no callback set 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.384+0000 7f2a55ffb640 10 monclient: dump: 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.405 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a67875640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.3:6790/0 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 --> v1:172.21.15.3:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2a40000f90 con 0x7f2a6009db30 2024-05-21T17:23:45.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f2a5c004060 con 0x7f2a6009db30 2024-05-21T17:23:45.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f2a5c045d40 con 0x7f2a6009db30 2024-05-21T17:23:45.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 --> v1:172.21.15.203:6808/2732955435 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a440434d0 con 0x7f2a4403f860 2024-05-21T17:23:45.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (unknown 3512546819 0 0) 0x7f2a5c0460c0 con 0x7f2a6009db30 2024-05-21T17:23:45.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a55ffb640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-05-21T17:23:45.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a64e10640 10 monclient: get_auth_request con 0x7f2a44038c80 auth_method 0 2024-05-21T17:23:45.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a65e12640 10 monclient: get_auth_request con 0x7f2a4403f860 auth_method 0 2024-05-21T17:23:45.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.388+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== osd.5 v1:172.21.15.203:6808/2732955435 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a440434d0 con 0x7f2a4403f860 2024-05-21T17:23:45.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 --> v1:172.21.15.203:6808/2732955435 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2a40002ce0 con 0x7f2a4403f860 2024-05-21T17:23:45.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a55ffb640 1 -- 172.21.15.3:0/3378800290 <== osd.5 v1:172.21.15.203:6808/2732955435 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f2a40002ce0 con 0x7f2a4403f860 2024-05-21T17:23:45.408 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-21T17:23:45.408 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-21T17:23:45.408 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-21T17:23:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 >> v1:172.21.15.203:6808/2732955435 conn(0x7f2a4403f860 legacy=0x7f2a44041c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 >> v1:172.21.15.203:6812/343827626 conn(0x7f2a44038c80 legacy=0x7f2a4403b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 10 monclient: shutdown 2024-05-21T17:23:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 >> v1:172.21.15.3:6790/0 conn(0x7f2a6009db30 legacy=0x7f2a60135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a65e12640 1 -- 172.21.15.3:0/3378800290 reap_dead start 2024-05-21T17:23:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 shutdown_connections 2024-05-21T17:23:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 >> 172.21.15.3:0/3378800290 conn(0x7f2a600a7ea0 msgr2=0x7f2a600a9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 shutdown_connections 2024-05-21T17:23:45.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.404+0000 7f2a67875640 1 -- 172.21.15.3:0/3378800290 wait complete. 2024-05-21T17:23:45.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-21T17:23:45.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient: get_monmap_and_config 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient: build_initial_monmap 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient: monmap: 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:45.536307+0000 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:45.536307+0000 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:45.535 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:45.536 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:45.536 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:45.536 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:45.536 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:45.536 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:45.536 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:45.536 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:45.536 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 1 Processor -- start 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 1 -- start start 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient: init 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient: _add_conns ranks=[5,7,1,8,0,6,3,4,2] 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient(hunting): picked mon.noname-h con 0x7f522809a4d0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient(hunting): picked mon.noname-i con 0x7f5228136d60 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient(hunting): picked mon.noname-f con 0x7f52281371a0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5228076c90 con 0x7f52281371a0 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f522809ea40 con 0x7f522809a4d0 2024-05-21T17:23:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f522809ed20 con 0x7f5228136d60 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient(hunting): authenticate will time out at 1092.693868s 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522659c640 1 --1- >> v1:172.21.15.203:6792/0 conn(0x7f5228136d60 0x7f522813ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6792/0 says I am v1:172.21.15.3:36308/0 (socket says 172.21.15.3:36308) 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522659c640 1 -- 172.21.15.3:0/2482611412 learned_addr learned my addr 172.21.15.3:0/2482611412 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936316393 0 0) 0x7f522809ed20 con 0x7f5228136d60 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): my global_id is 4813 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5208003530 con 0x7f5228136d60 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2676460093 0 0) 0x7f522809ea40 con 0x7f522809a4d0 2024-05-21T17:23:45.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): my global_id is 4829 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f522809ed20 con 0x7f522809a4d0 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2134676919 0 0) 0x7f5228076c90 con 0x7f52281371a0 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient(hunting): my global_id is 134281 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f522809ea40 con 0x7f52281371a0 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 <== mon.7 v1:172.21.15.203:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 977631877 0 0) 0x7f5208003530 con 0x7f5228136d60 2024-05-21T17:23:45.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 >> v1:172.21.15.203:6791/0 conn(0x7f522809a4d0 legacy=0x7f522809a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 >> v1:172.21.15.203:6789/0 conn(0x7f52281371a0 legacy=0x7f522813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: _finish_hunting 0 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 monclient: found mon.noname-i 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.203:6792/0 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 --> v1:172.21.15.203:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522809f000 con 0x7f5228136d60 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: _finish_auth 0 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: _check_auth_tickets 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 5 monclient: authenticate success, global_id 4813 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 <== mon.7 v1:172.21.15.203:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5220001bc0 con 0x7f5228136d60 2024-05-21T17:23:45.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: dump: 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 <== mon.7 v1:172.21.15.203:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5220001de0 con 0x7f5228136d60 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f522d115640 4 set_mon_vals no callback set 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 1 -- 172.21.15.3:0/2482611412 <== mon.7 v1:172.21.15.203:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5220004f70 con 0x7f5228136d60 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.532+0000 7f5225d9b640 10 monclient: dump: 2024-05-21T17:23:45.548 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.549 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: shutdown 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- 172.21.15.3:0/2482611412 >> v1:172.21.15.203:6792/0 conn(0x7f5228136d60 legacy=0x7f522813ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- 172.21.15.3:0/2482611412 shutdown_connections 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- 172.21.15.3:0/2482611412 >> 172.21.15.3:0/2482611412 conn(0x7f52280a3f10 msgr2=0x7f52280a4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- 172.21.15.3:0/2482611412 shutdown_connections 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- 172.21.15.3:0/2482611412 wait complete. 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: build_initial_monmap 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: monmap: 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:45.550 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:45.551 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 Processor -- start 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- start start 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: init 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: _add_conns ranks=[1,4,5,8,7,0,3,2,6] 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient(hunting): picked mon.noname-f con 0x7f522809a4d0 addr v1:172.21.15.203:6789/0 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient(hunting): picked mon.noname-c con 0x7f5228136d60 addr v1:172.21.15.3:6791/0 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient(hunting): picked mon.noname-h con 0x7f52281371a0 addr v1:172.21.15.203:6791/0 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f522809b980 con 0x7f522809a4d0 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- --> v1:172.21.15.3:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5228071180 con 0x7f5228136d60 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5228070ea0 con 0x7f52281371a0 2024-05-21T17:23:45.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient(hunting): authenticate will time out at 1092.697769s 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522659c640 1 --1- >> v1:172.21.15.3:6791/0 conn(0x7f5228136d60 0x7f52281d2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6791/0 says I am v1:172.21.15.3:55004/0 (socket says 172.21.15.3:55004) 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522659c640 1 -- 172.21.15.3:0/3398415237 learned_addr learned my addr 172.21.15.3:0/3398415237 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581665604 0 0) 0x7f5228071180 con 0x7f5228136d60 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): my global_id is 4747 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.3:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f520c003530 con 0x7f5228136d60 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2936784920 0 0) 0x7f520c003530 con 0x7f5228136d60 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.3:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5228071180 con 0x7f5228136d60 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5220004eb0 con 0x7f5228136d60 2024-05-21T17:23:45.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): dump: 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3378350933 0 0) 0x7f522809b980 con 0x7f522809a4d0 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): my global_id is 134290 2024-05-21T17:23:45.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f520c003530 con 0x7f522809a4d0 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.5 v1:172.21.15.203:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 685354093 0 0) 0x7f5228070ea0 con 0x7f52281371a0 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient(hunting): my global_id is 4838 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.203:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f522809b980 con 0x7f52281371a0 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3333606463 0 0) 0x7f5228071180 con 0x7f5228136d60 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 >> v1:172.21.15.203:6791/0 conn(0x7f52281371a0 legacy=0x7f52281d6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 >> v1:172.21.15.203:6789/0 conn(0x7f522809a4d0 legacy=0x7f52281cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 monclient: found mon.c 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.3:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522809c4c0 con 0x7f5228136d60 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient: _finish_auth 0 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 5 monclient: authenticate success, global_id 4747 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: _renew_subs 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.3:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5228099a10 con 0x7f5228136d60 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: _renew_subs 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.3:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f52281ce470 con 0x7f5228136d60 2024-05-21T17:23:45.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5220001580 con 0x7f5228136d60 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f5220001ce0 con 0x7f5228136d60 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522559a640 4 set_mon_vals no callback set 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 10 monclient: dump: 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.558 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6791/0 2024-05-21T17:23:45.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.3:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5200005090 con 0x7f5228136d60 2024-05-21T17:23:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f5220010550 con 0x7f5228136d60 2024-05-21T17:23:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.536+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f5220046020 con 0x7f5228136d60 2024-05-21T17:23:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.540+0000 7f5226d9d640 10 monclient: get_auth_request con 0x7f520c038ef0 auth_method 0 2024-05-21T17:23:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.544+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mon.4 v1:172.21.15.3:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f52200161a0 con 0x7f5228136d60 2024-05-21T17:23:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.544+0000 7f521f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.544+0000 7f521f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.704+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f5200000ba0 con 0x7f520c038ef0 2024-05-21T17:23:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f521f7fe640 1 -- 172.21.15.3:0/3398415237 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7f5200000ba0 con 0x7f520c038ef0 2024-05-21T17:23:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 5 to deep-scrub 2024-05-21T17:23:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 >> v1:172.21.15.203:6812/343827626 conn(0x7f520c038ef0 legacy=0x7f520c03b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f522d115640 10 monclient: shutdown 2024-05-21T17:23:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f522d115640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 >> v1:172.21.15.3:6791/0 conn(0x7f5228136d60 legacy=0x7f52281d2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 shutdown_connections 2024-05-21T17:23:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 >> 172.21.15.3:0/3398415237 conn(0x7f52280a3f10 msgr2=0x7f5228075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 shutdown_connections 2024-05-21T17:23:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.708+0000 7f522d115640 1 -- 172.21.15.3:0/3398415237 wait complete. 2024-05-21T17:23:45.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:23:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient: get_monmap_and_config 2024-05-21T17:23:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient: build_initial_monmap 2024-05-21T17:23:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient: monmap: 2024-05-21T17:23:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:23:45.849800+0000 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:23:45.849800+0000 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:45.850 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:45.850 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 1 Processor -- start 2024-05-21T17:23:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 1 -- start start 2024-05-21T17:23:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient: init 2024-05-21T17:23:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient: _add_conns ranks=[0,3,8,6,4,5,7,2,1] 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient(hunting): picked mon.noname-a con 0x7f30980a1d00 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient(hunting): picked mon.noname-g con 0x7f30980a16c0 addr v1:172.21.15.203:6790/0 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient(hunting): picked mon.noname-e con 0x7f309813d050 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30980a3220 con 0x7f30980a1d00 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30980a2c60 con 0x7f30980a16c0 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30980a1aa0 con 0x7f309813d050 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f30a0083640 10 monclient(hunting): authenticate will time out at 1093.007216s 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309e620640 1 --1- >> v1:172.21.15.3:6793/0 conn(0x7f309813d050 0x7f309813f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6793/0 says I am v1:172.21.15.3:55122/0 (socket says 172.21.15.3:55122) 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309e620640 1 -- 172.21.15.3:0/3568268638 learned_addr learned my addr 172.21.15.3:0/3568268638 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309de1f640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f30980a1d00 0x7f30980a20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38578/0 (socket says 172.21.15.3:38578) 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363060257 0 0) 0x7f30980a1aa0 con 0x7f309813d050 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 10 monclient(hunting): my global_id is 4805 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3084003530 con 0x7f309813d050 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971890225 0 0) 0x7f30980a3220 con 0x7f30980a1d00 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 10 monclient(hunting): my global_id is 4779 2024-05-21T17:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30980a1aa0 con 0x7f30980a1d00 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.844+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1886411878 0 0) 0x7f3084003530 con 0x7f309813d050 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 >> v1:172.21.15.203:6790/0 conn(0x7f30980a16c0 legacy=0x7f309813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 >> v1:172.21.15.3:6789/0 conn(0x7f30980a1d00 legacy=0x7f30980a20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: _finish_hunting 0 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 1 monclient: found mon.noname-e 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.3:6793/0 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 --> v1:172.21.15.3:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30980a1070 con 0x7f309813d050 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: _finish_auth 0 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: _check_auth_tickets 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3090001bc0 con 0x7f309813d050 2024-05-21T17:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 5 monclient: authenticate success, global_id 4805 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: dump: 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 <== mon.8 v1:172.21.15.3:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3090001de0 con 0x7f309813d050 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: get_monmap_and_config success 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 1 -- 172.21.15.3:0/3568268638 <== mon.8 v1:172.21.15.3:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3090004f70 con 0x7f309813d050 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 4 set_mon_vals no callback set 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309ce1d640 10 monclient: dump: 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: shutdown 2024-05-21T17:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- 172.21.15.3:0/3568268638 >> v1:172.21.15.3:6793/0 conn(0x7f309813d050 legacy=0x7f309813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- 172.21.15.3:0/3568268638 shutdown_connections 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- 172.21.15.3:0/3568268638 >> 172.21.15.3:0/3568268638 conn(0x7f30980a72c0 msgr2=0x7f30980a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- 172.21.15.3:0/3568268638 shutdown_connections 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- 172.21.15.3:0/3568268638 wait complete. 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: build_initial_monmap 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: monmap: 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 Processor -- start 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- start start 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: init 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: _reopen_session rank -1 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: _add_conns ranks=[7,8,0,5,2,3,6,1,4] 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient(hunting): picked mon.noname-i con 0x7f30980a16c0 addr v1:172.21.15.203:6792/0 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient(hunting): picked mon.noname-e con 0x7f30980a1d00 addr v1:172.21.15.3:6793/0 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient(hunting): picked mon.noname-a con 0x7f309813d050 addr v1:172.21.15.3:6789/0 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f309809efe0 con 0x7f309813d050 2024-05-21T17:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f309809f4f0 con 0x7f30980a16c0 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30980a0030 con 0x7f30980a1d00 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient(hunting): _renew_subs 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient(hunting): authenticate will time out at 1093.010220s 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309e620640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f309813d050 0x7f30981d5cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38586/0 (socket says 172.21.15.3:38586) 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f309e620640 1 -- 172.21.15.3:0/678492900 learned_addr learned my addr 172.21.15.3:0/678492900 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2520399031 0 0) 0x7f309809efe0 con 0x7f309813d050 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): my global_id is 4788 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f307c003530 con 0x7f309813d050 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827051578 0 0) 0x7f30980a0030 con 0x7f30980a1d00 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): my global_id is 4814 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f309809efe0 con 0x7f30980a1d00 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1970789712 0 0) 0x7f307c003530 con 0x7f309813d050 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30980a0030 con 0x7f309813d050 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3090001f90 con 0x7f309813d050 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): dump: 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.8 v1:172.21.15.3:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2753383413 0 0) 0x7f309809efe0 con 0x7f30980a1d00 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.3:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f307c003530 con 0x7f30980a1d00 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.8 v1:172.21.15.3:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3088001aa0 con 0x7f30980a1d00 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): dump: 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4278573650 0 0) 0x7f309809f4f0 con 0x7f30980a16c0 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _negotiate 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient(hunting): my global_id is 4822 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f309809efe0 con 0x7f30980a16c0 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 796314898 0 0) 0x7f30980a0030 con 0x7f309813d050 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 >> v1:172.21.15.3:6793/0 conn(0x7f30980a1d00 legacy=0x7f309809e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 >> v1:172.21.15.203:6792/0 conn(0x7f30980a16c0 legacy=0x7f309809e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient: _finish_hunting 0 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 monclient: found mon.a 2024-05-21T17:23:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309813bad0 con 0x7f309813d050 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient: _finish_auth 0 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient: _check_auth_tickets 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 5 monclient: authenticate success, global_id 4788 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: _renew_subs 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3098072960 con 0x7f309813d050 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: _renew_subs 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f30981d73f0 con 0x7f309813d050 2024-05-21T17:23:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3090001500 con 0x7f309813d050 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient: handle_config config(0 keys) 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f3090001ca0 con 0x7f309813d050 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308ffff640 4 set_mon_vals no callback set 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f308e7fc640 10 monclient: dump: 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:23:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:23:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.848+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3080005090 con 0x7f309813d050 2024-05-21T17:23:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.852+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f3090010480 con 0x7f309813d050 2024-05-21T17:23:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.852+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f3090045da0 con 0x7f309813d050 2024-05-21T17:23:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.852+0000 7f309de1f640 10 monclient: get_auth_request con 0x7f307c0393b0 auth_method 0 2024-05-21T17:23:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.856+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f3090016120 con 0x7f309813d050 2024-05-21T17:23:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.856+0000 7f308e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:23:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:45.856+0000 7f308e7fc640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:23:46.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.016+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3080000ba0 con 0x7f307c0393b0 2024-05-21T17:23:46.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f308e7fc640 1 -- 172.21.15.3:0/678492900 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33795 (unknown 2965378022 0 447182020) 0x7f3080000ba0 con 0x7f307c0393b0 2024-05-21T17:23:46.023 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:23:46.023 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:23:46.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 >> v1:172.21.15.203:6812/343827626 conn(0x7f307c0393b0 legacy=0x7f307c03b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:46.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f30a0083640 10 monclient: shutdown 2024-05-21T17:23:46.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f30a0083640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:23:46.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 >> v1:172.21.15.3:6789/0 conn(0x7f309813d050 legacy=0x7f30981d5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:23:46.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 shutdown_connections 2024-05-21T17:23:46.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 >> 172.21.15.3:0/678492900 conn(0x7f30980a72c0 msgr2=0x7f3098140270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:23:46.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 shutdown_connections 2024-05-21T17:23:46.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:23:46.020+0000 7f30a0083640 1 -- 172.21.15.3:0/678492900 wait complete. 2024-05-21T17:23:46.039 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-05-21T17:23:45.094995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163352,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067688,"statfs":{"total":579820584960,"available":579653312512,"internally_reserved":0,"allocated":1236992,"data_stored":1439006,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003104"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.377367+0000","last_change":"2024-05-21T17:21:38.226793+0000","last_active":"2024-05-21T17:22:04.377367+0000","last_peered":"2024-05-21T17:22:04.377367+0000","last_clean":"2024-05-21T17:22:04.377367+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:22:04.377367+0000","last_undegraded":"2024-05-21T17:22:04.377367+0000","last_fullsized":"2024-05-21T17:22:04.377367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T21:23:30.360529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.371254+0000","last_change":"2024-05-21T17:21:38.830568+0000","last_active":"2024-05-21T17:22:04.371254+0000","last_peered":"2024-05-21T17:22:04.371254+0000","last_clean":"2024-05-21T17:22:04.371254+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:22:04.371254+0000","last_undegraded":"2024-05-21T17:22:04.371254+0000","last_fullsized":"2024-05-21T17:22:04.371254+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T02:04:57.235930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.731426+0000","last_change":"2024-05-21T17:21:38.216293+0000","last_active":"2024-05-21T17:22:04.731426+0000","last_peered":"2024-05-21T17:22:04.731426+0000","last_clean":"2024-05-21T17:22:04.731426+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:22:04.731426+0000","last_undegraded":"2024-05-21T17:22:04.731426+0000","last_fullsized":"2024-05-21T17:22:04.731426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:53:31.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.374521+0000","last_change":"2024-05-21T17:21:38.219369+0000","last_active":"2024-05-21T17:22:04.374521+0000","last_peered":"2024-05-21T17:22:04.374521+0000","last_clean":"2024-05-21T17:22:04.374521+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:22:04.374521+0000","last_undegraded":"2024-05-21T17:22:04.374521+0000","last_fullsized":"2024-05-21T17:22:04.374521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:24:49.428951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'72","reported_seq":115,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373833+0000","last_change":"2024-05-21T17:21:38.221127+0000","last_active":"2024-05-21T17:22:04.373833+0000","last_peered":"2024-05-21T17:22:04.373833+0000","last_clean":"2024-05-21T17:22:04.373833+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:22:04.373833+0000","last_undegraded":"2024-05-21T17:22:04.373833+0000","last_fullsized":"2024-05-21T17:22:04.373833+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:12:45.986667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.370522+0000","last_change":"2024-05-21T17:21:38.219803+0000","last_active":"2024-05-21T17:22:04.370522+0000","last_peered":"2024-05-21T17:22:04.370522+0000","last_clean":"2024-05-21T17:22:04.370522+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:22:04.370522+0000","last_undegraded":"2024-05-21T17:22:04.370522+0000","last_fullsized":"2024-05-21T17:22:04.370522+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:52:49.624765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.729732+0000","last_change":"2024-05-21T17:21:38.829468+0000","last_active":"2024-05-21T17:22:04.729732+0000","last_peered":"2024-05-21T17:22:04.729732+0000","last_clean":"2024-05-21T17:22:04.729732+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:22:04.729732+0000","last_undegraded":"2024-05-21T17:22:04.729732+0000","last_fullsized":"2024-05-21T17:22:04.729732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:07:14.487479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.373326+0000","last_change":"2024-05-21T17:21:38.830884+0000","last_active":"2024-05-21T17:22:04.373326+0000","last_peered":"2024-05-21T17:22:04.373326+0000","last_clean":"2024-05-21T17:22:04.373326+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:22:04.373326+0000","last_undegraded":"2024-05-21T17:22:04.373326+0000","last_fullsized":"2024-05-21T17:22:04.373326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:36:41.643111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:22:04.374049+0000","last_change":"2024-05-21T17:21:38.829272+0000","last_active":"2024-05-21T17:22:04.374049+0000","last_peered":"2024-05-21T17:22:04.374049+0000","last_clean":"2024-05-21T17:22:04.374049+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:22:04.374049+0000","last_undegraded":"2024-05-21T17:22:04.374049+0000","last_fullsized":"2024-05-21T17:22:04.374049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:21:37.183217+0000","last_clean_scrub_stamp":"2024-05-21T17:21:37.183217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:09:03.800931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640285,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640283,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640285,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:23:46.043 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.043 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.043 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.043 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.043 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.043 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.043 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.043 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.044 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-21T17:21:37.183217+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=21, tm_sec=37, tm_wday=1, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=21, tm_hour=17, tm_min=23, tm_sec=41, tm_wday=1, tm_yday=142, tm_isdst=0) 2024-05-21T17:23:46.044 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-21T17:24:06.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-21T17:24:06.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient: get_monmap_and_config 2024-05-21T17:24:06.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient: build_initial_monmap 2024-05-21T17:24:06.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient: monmap: 2024-05-21T17:24:06.187 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:24:06.188753+0000 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:24:06.188753+0000 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:24:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:24:06.189 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:24:06.189 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:24:06.189 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 Processor -- start 2024-05-21T17:24:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 -- start start 2024-05-21T17:24:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient: init 2024-05-21T17:24:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient: _reopen_session rank -1 2024-05-21T17:24:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient: _add_conns ranks=[0,8,5,6,4,7,2,3,1] 2024-05-21T17:24:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient(hunting): picked mon.noname-a con 0x7fee9009d570 addr v1:172.21.15.3:6789/0 2024-05-21T17:24:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient(hunting): picked mon.noname-e con 0x7fee9009e060 addr v1:172.21.15.3:6793/0 2024-05-21T17:24:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient(hunting): picked mon.noname-h con 0x7fee9009d950 addr v1:172.21.15.203:6791/0 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee9009c0b0 con 0x7fee9009d570 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 -- --> v1:172.21.15.203:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee900a4f10 con 0x7fee9009d950 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 -- --> v1:172.21.15.3:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee900a3d20 con 0x7fee9009e060 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient(hunting): _renew_subs 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient(hunting): authenticate will time out at 1113.346129s 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee94d7e640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fee9009d570 0x7fee900a2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:45410/0 (socket says 172.21.15.3:45410) 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee94d7e640 1 -- 172.21.15.3:0/376239720 learned_addr learned my addr 172.21.15.3:0/376239720 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1328027301 0 0) 0x7fee9009c0b0 con 0x7fee9009d570 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient(hunting): my global_id is 4797 2024-05-21T17:24:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee78003530 con 0x7fee9009d570 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 <== mon.8 v1:172.21.15.3:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2272479442 0 0) 0x7fee900a3d20 con 0x7fee9009e060 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient(hunting): _negotiate 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient(hunting): my global_id is 4823 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 --> v1:172.21.15.3:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee9009c0b0 con 0x7fee9009e060 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1940449054 0 0) 0x7fee78003530 con 0x7fee9009d570 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 >> v1:172.21.15.3:6793/0 conn(0x7fee9009e060 legacy=0x7fee900a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 >> v1:172.21.15.203:6791/0 conn(0x7fee9009d950 legacy=0x7fee90141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: _finish_hunting 0 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 monclient: found mon.noname-a 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:24:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9009b7a0 con 0x7fee9009d570 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: _finish_auth 0 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: _check_auth_tickets 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fee80001840 con 0x7fee9009d570 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: dump: 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 5 monclient: authenticate success, global_id 4797 2024-05-21T17:24:06.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee80001a60 con 0x7fee9009d570 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: handle_config config(0 keys) 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 1 -- 172.21.15.3:0/376239720 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fee80004bf0 con 0x7fee9009d570 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient: get_monmap_and_config success 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 4 set_mon_vals no callback set 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee8f7fe640 10 monclient: dump: 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 10 monclient: shutdown 2024-05-21T17:24:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 -- 172.21.15.3:0/376239720 >> v1:172.21.15.3:6789/0 conn(0x7fee9009d570 legacy=0x7fee900a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 -- 172.21.15.3:0/376239720 shutdown_connections 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 -- 172.21.15.3:0/376239720 >> 172.21.15.3:0/376239720 conn(0x7fee900a3ff0 msgr2=0x7fee900a4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.184+0000 7fee96fe2640 1 -- 172.21.15.3:0/376239720 shutdown_connections 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 -- 172.21.15.3:0/376239720 wait complete. 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: build_initial_monmap 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: monmap: 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:24:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 Processor -- start 2024-05-21T17:24:06.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 -- start start 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: init 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: _reopen_session rank -1 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: _add_conns ranks=[3,0,1,8,4,7,6,5,2] 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient(hunting): picked mon.noname-g con 0x7fee9009d570 addr v1:172.21.15.203:6790/0 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient(hunting): picked mon.noname-a con 0x7fee9009d950 addr v1:172.21.15.3:6789/0 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient(hunting): picked mon.noname-f con 0x7fee9009e060 addr v1:172.21.15.203:6789/0 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee900936a0 con 0x7fee9009d950 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee90094570 con 0x7fee9009e060 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 -- --> v1:172.21.15.203:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee90094850 con 0x7fee9009d570 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient(hunting): _renew_subs 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient(hunting): authenticate will time out at 1113.349470s 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8ffff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fee9009d950 0x7fee901d2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:45424/0 (socket says 172.21.15.3:45424) 2024-05-21T17:24:06.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8ffff640 1 -- 172.21.15.3:0/161367239 learned_addr learned my addr 172.21.15.3:0/161367239 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268479360 0 0) 0x7fee900936a0 con 0x7fee9009d950 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient(hunting): _negotiate 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient(hunting): my global_id is 4806 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee7c003530 con 0x7fee9009d950 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 194081047 0 0) 0x7fee7c003530 con 0x7fee9009d950 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee900936a0 con 0x7fee9009d950 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fee84001b00 con 0x7fee9009d950 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:24:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient(hunting): dump: 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2807240276 0 0) 0x7fee900936a0 con 0x7fee9009d950 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 >> v1:172.21.15.203:6790/0 conn(0x7fee9009d570 legacy=0x7fee90097330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 >> v1:172.21.15.203:6789/0 conn(0x7fee9009e060 legacy=0x7fee901d6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient: _finish_hunting 0 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 monclient: found mon.a 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee90094b30 con 0x7fee9009d950 2024-05-21T17:24:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient: _finish_auth 0 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient: _check_auth_tickets 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 5 monclient: authenticate success, global_id 4806 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: _renew_subs 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee90092af0 con 0x7fee9009d950 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: _renew_subs 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fee90097c10 con 0x7fee9009d950 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee840038b0 con 0x7fee9009d950 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient: handle_config config(0 keys) 2024-05-21T17:24:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7fee84003e20 con 0x7fee9009d950 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8effd640 4 set_mon_vals no callback set 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 10 monclient: dump: 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7fee84003fc0 con 0x7fee9009d950 2024-05-21T17:24:06.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee70005090 con 0x7fee9009d950 2024-05-21T17:24:06.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7fee84045be0 con 0x7fee9009d950 2024-05-21T17:24:06.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.188+0000 7fee94d7e640 10 monclient: get_auth_request con 0x7fee7c038d00 auth_method 0 2024-05-21T17:24:06.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.196+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7fee84015c40 con 0x7fee9009d950 2024-05-21T17:24:06.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.196+0000 7fee8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:24:06.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.196+0000 7fee8d7fa640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:24:06.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.356+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 --> v1:172.21.15.203:6812/343827626 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fee70000ba0 con 0x7fee7c038d00 2024-05-21T17:24:06.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.360+0000 7fee8d7fa640 1 -- 172.21.15.3:0/161367239 <== mgr.4127 v1:172.21.15.203:6812/343827626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33799 (unknown 2965378022 0 4025814680) 0x7fee70000ba0 con 0x7fee7c038d00 2024-05-21T17:24:06.365 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-21T17:24:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-21T17:24:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.360+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 >> v1:172.21.15.203:6812/343827626 conn(0x7fee7c038d00 legacy=0x7fee7c03b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.360+0000 7fee96fe2640 10 monclient: shutdown 2024-05-21T17:24:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.360+0000 7fee96fe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:24:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.360+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 >> v1:172.21.15.3:6789/0 conn(0x7fee9009d950 legacy=0x7fee901d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.364+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 shutdown_connections 2024-05-21T17:24:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.364+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 >> 172.21.15.3:0/161367239 conn(0x7fee900a3ff0 msgr2=0x7fee9009e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:24:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.364+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 shutdown_connections 2024-05-21T17:24:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.364+0000 7fee96fe2640 1 -- 172.21.15.3:0/161367239 wait complete. 2024-05-21T17:24:06.383 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-05-21T17:24:05.100349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163304,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067736,"statfs":{"total":579820584960,"available":579653361664,"internally_reserved":0,"allocated":1236992,"data_stored":1439006,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003235"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:46.073448+0000","last_change":"2024-05-21T17:23:46.073448+0000","last_active":"2024-05-21T17:23:46.073448+0000","last_peered":"2024-05-21T17:23:46.073448+0000","last_clean":"2024-05-21T17:23:46.073448+0000","last_became_active":"2024-05-21T17:21:38.225888+0000","last_became_peered":"2024-05-21T17:21:38.225888+0000","last_unstale":"2024-05-21T17:23:46.073448+0000","last_undegraded":"2024-05-21T17:23:46.073448+0000","last_fullsized":"2024-05-21T17:23:46.073448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:23:46.073207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:23:46.073207+0000","last_clean_scrub_stamp":"2024-05-21T17:23:46.073207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T23:25:08.563375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:47.530790+0000","last_change":"2024-05-21T17:23:47.530846+0000","last_active":"2024-05-21T17:23:47.530790+0000","last_peered":"2024-05-21T17:23:47.530790+0000","last_clean":"2024-05-21T17:23:47.530790+0000","last_became_active":"2024-05-21T17:21:38.830145+0000","last_became_peered":"2024-05-21T17:21:38.830145+0000","last_unstale":"2024-05-21T17:23:47.530790+0000","last_undegraded":"2024-05-21T17:23:47.530790+0000","last_fullsized":"2024-05-21T17:23:47.530790+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:23:47.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:23:47.530612+0000","last_clean_scrub_stamp":"2024-05-21T17:23:47.530612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T05:23:44.219124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:46.087603+0000","last_change":"2024-05-21T17:23:46.087603+0000","last_active":"2024-05-21T17:23:46.087603+0000","last_peered":"2024-05-21T17:23:46.087603+0000","last_clean":"2024-05-21T17:23:46.087603+0000","last_became_active":"2024-05-21T17:21:38.215629+0000","last_became_peered":"2024-05-21T17:21:38.215629+0000","last_unstale":"2024-05-21T17:23:46.087603+0000","last_undegraded":"2024-05-21T17:23:46.087603+0000","last_fullsized":"2024-05-21T17:23:46.087603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:23:46.087258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:23:46.087258+0000","last_clean_scrub_stamp":"2024-05-21T17:23:46.087258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T05:15:52.770332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:44.777946+0000","last_change":"2024-05-21T17:23:44.777946+0000","last_active":"2024-05-21T17:23:44.777946+0000","last_peered":"2024-05-21T17:23:44.777946+0000","last_clean":"2024-05-21T17:23:44.777946+0000","last_became_active":"2024-05-21T17:21:38.218996+0000","last_became_peered":"2024-05-21T17:21:38.218996+0000","last_unstale":"2024-05-21T17:23:44.777946+0000","last_undegraded":"2024-05-21T17:23:44.777946+0000","last_fullsized":"2024-05-21T17:23:44.777946+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:23:44.777866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:23:44.777866+0000","last_clean_scrub_stamp":"2024-05-21T17:23:44.777866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T21:27:57.068343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"21'74","reported_seq":126,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:45.561260+0000","last_change":"2024-05-21T17:23:45.561328+0000","last_active":"2024-05-21T17:23:45.561260+0000","last_peered":"2024-05-21T17:23:45.561260+0000","last_clean":"2024-05-21T17:23:45.561260+0000","last_became_active":"2024-05-21T17:21:38.220585+0000","last_became_peered":"2024-05-21T17:21:38.220585+0000","last_unstale":"2024-05-21T17:23:45.561260+0000","last_undegraded":"2024-05-21T17:23:45.561260+0000","last_fullsized":"2024-05-21T17:23:45.561260+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'74","last_scrub_stamp":"2024-05-21T17:23:45.561084+0000","last_deep_scrub":"21'74","last_deep_scrub_stamp":"2024-05-21T17:23:45.561084+0000","last_clean_scrub_stamp":"2024-05-21T17:23:45.561084+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T03:18:50.349915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:45.757727+0000","last_change":"2024-05-21T17:23:45.757727+0000","last_active":"2024-05-21T17:23:45.757727+0000","last_peered":"2024-05-21T17:23:45.757727+0000","last_clean":"2024-05-21T17:23:45.757727+0000","last_became_active":"2024-05-21T17:21:38.218941+0000","last_became_peered":"2024-05-21T17:21:38.218941+0000","last_unstale":"2024-05-21T17:23:45.757727+0000","last_undegraded":"2024-05-21T17:23:45.757727+0000","last_fullsized":"2024-05-21T17:23:45.757727+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:23:45.757623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:23:45.757623+0000","last_clean_scrub_stamp":"2024-05-21T17:23:45.757623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T20:28:32.925815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:47.122194+0000","last_change":"2024-05-21T17:23:47.122247+0000","last_active":"2024-05-21T17:23:47.122194+0000","last_peered":"2024-05-21T17:23:47.122194+0000","last_clean":"2024-05-21T17:23:47.122194+0000","last_became_active":"2024-05-21T17:21:38.829043+0000","last_became_peered":"2024-05-21T17:21:38.829043+0000","last_unstale":"2024-05-21T17:23:47.122194+0000","last_undegraded":"2024-05-21T17:23:47.122194+0000","last_fullsized":"2024-05-21T17:23:47.122194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:23:47.122022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:23:47.122022+0000","last_clean_scrub_stamp":"2024-05-21T17:23:47.122022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T22:55:06.529412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:47.082132+0000","last_change":"2024-05-21T17:23:47.082218+0000","last_active":"2024-05-21T17:23:47.082132+0000","last_peered":"2024-05-21T17:23:47.082132+0000","last_clean":"2024-05-21T17:23:47.082132+0000","last_became_active":"2024-05-21T17:21:38.829974+0000","last_became_peered":"2024-05-21T17:21:38.829974+0000","last_unstale":"2024-05-21T17:23:47.082132+0000","last_undegraded":"2024-05-21T17:23:47.082132+0000","last_fullsized":"2024-05-21T17:23:47.082132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:23:47.081913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:23:47.081913+0000","last_clean_scrub_stamp":"2024-05-21T17:23:47.081913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T23:05:59.459630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-21T17:23:46.580407+0000","last_change":"2024-05-21T17:23:46.580465+0000","last_active":"2024-05-21T17:23:46.580407+0000","last_peered":"2024-05-21T17:23:46.580407+0000","last_clean":"2024-05-21T17:23:46.580407+0000","last_became_active":"2024-05-21T17:21:38.828746+0000","last_became_peered":"2024-05-21T17:21:38.828746+0000","last_unstale":"2024-05-21T17:23:46.580407+0000","last_undegraded":"2024-05-21T17:23:46.580407+0000","last_fullsized":"2024-05-21T17:23:46.580407+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T17:23:46.580207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T17:23:46.580207+0000","last_clean_scrub_stamp":"2024-05-21T17:23:46.580207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T18:02:49.126666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640289,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640287,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640288,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":592453,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640289,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":188416,"data_stored":63525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T17:24:06.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-21T17:24:06.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.504+0000 7f4f2cd4d640 10 monclient: get_monmap_and_config 2024-05-21T17:24:06.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.504+0000 7f4f2cd4d640 10 monclient: build_initial_monmap 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.504+0000 7f4f2cd4d640 10 monclient: monmap: 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:24:06.512070+0000 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:24:06.512070+0000 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:24:06.511 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:24:06.512 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:24:06.512 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:24:06.512 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 Processor -- start 2024-05-21T17:24:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- start start 2024-05-21T17:24:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient: init 2024-05-21T17:24:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient: _reopen_session rank -1 2024-05-21T17:24:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient: _add_conns ranks=[7,1,0,8,6,4,2,3,5] 2024-05-21T17:24:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient(hunting): picked mon.noname-i con 0x7f4f28135c20 addr v1:172.21.15.203:6792/0 2024-05-21T17:24:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient(hunting): picked mon.noname-f con 0x7f4f28139bf0 addr v1:172.21.15.203:6789/0 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f2813d700 addr v1:172.21.15.3:6789/0 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f280a38e0 con 0x7f4f2813d700 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f280a1320 con 0x7f4f28139bf0 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f280a2ee0 con 0x7f4f28135c20 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient(hunting): _renew_subs 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient(hunting): authenticate will time out at 1113.669617s 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f26d9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4f2813d700 0x7f4f2813faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:45430/0 (socket says 172.21.15.3:45430) 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f26d9d640 1 -- 172.21.15.3:0/2329474100 learned_addr learned my addr 172.21.15.3:0/2329474100 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1178915797 0 0) 0x7f4f280a38e0 con 0x7f4f2813d700 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient(hunting): _negotiate 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:24:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient(hunting): my global_id is 4815 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f14003530 con 0x7f4f2813d700 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1858233583 0 0) 0x7f4f14003530 con 0x7f4f2813d700 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 >> v1:172.21.15.203:6792/0 conn(0x7f4f28135c20 legacy=0x7f4f28136000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 >> v1:172.21.15.203:6789/0 conn(0x7f4f28139bf0 legacy=0x7f4f2813bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: _finish_hunting 0 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 monclient: found mon.noname-a 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2809db40 con 0x7f4f2813d700 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: _finish_auth 0 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: _check_auth_tickets 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:24:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4f20001bc0 con 0x7f4f2813d700 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 5 monclient: authenticate success, global_id 4815 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: dump: 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f20001de0 con 0x7f4f2813d700 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: handle_config config(0 keys) 2024-05-21T17:24:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient: get_monmap_and_config success 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 1 -- 172.21.15.3:0/2329474100 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4f200020d0 con 0x7f4f2813d700 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 4 set_mon_vals no callback set 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f25d9b640 10 monclient: dump: 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient: shutdown 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/2329474100 >> v1:172.21.15.3:6789/0 conn(0x7f4f2813d700 legacy=0x7f4f2813faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/2329474100 shutdown_connections 2024-05-21T17:24:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/2329474100 >> 172.21.15.3:0/2329474100 conn(0x7f4f280a3b90 msgr2=0x7f4f280a3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/2329474100 shutdown_connections 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/2329474100 wait complete. 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient: build_initial_monmap 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.508+0000 7f4f2cd4d640 10 monclient: monmap: 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.noname-f 2024-05-21T17:24:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.noname-g 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.noname-c 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.noname-h 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.noname-d 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.noname-i 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.noname-e 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 1 Processor -- start 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 1 -- start start 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: init 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: _reopen_session rank -1 2024-05-21T17:24:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: _add_conns ranks=[0,7,6,4,8,5,1,3,2] 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f28135c20 addr v1:172.21.15.3:6789/0 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient(hunting): picked mon.noname-i con 0x7f4f28139bf0 addr v1:172.21.15.203:6792/0 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient(hunting): picked mon.noname-d con 0x7f4f2813d700 addr v1:172.21.15.3:6792/0 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f28134ce0 con 0x7f4f28135c20 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 1 -- --> v1:172.21.15.3:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f2811fc90 con 0x7f4f2813d700 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 1 -- --> v1:172.21.15.203:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f28073f90 con 0x7f4f28139bf0 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient(hunting): _renew_subs 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient(hunting): authenticate will time out at 1113.672691s 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4f28135c20 0x7f4f281345d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:45444/0 (socket says 172.21.15.3:45444) 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2659c640 1 -- 172.21.15.3:0/4055932324 learned_addr learned my addr 172.21.15.3:0/4055932324 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076039134 0 0) 0x7f4f28134ce0 con 0x7f4f28135c20 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _negotiate 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:24:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): my global_id is 4824 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f08003530 con 0x7f4f28135c20 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.6 v1:172.21.15.3:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 337954043 0 0) 0x7f4f2811fc90 con 0x7f4f2813d700 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _negotiate 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): my global_id is 4587 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f28134ce0 con 0x7f4f2813d700 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4223132100 0 0) 0x7f4f08003530 con 0x7f4f28135c20 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f2811fc90 con 0x7f4f28135c20 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4f10001ae0 con 0x7f4f28135c20 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-05-21T17:24:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): dump: 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.529 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _finish_auth 0 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.7 v1:172.21.15.203:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161849324 0 0) 0x7f4f28073f90 con 0x7f4f28139bf0 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _negotiate 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _init_auth method 2 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): my global_id is 4840 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.203:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f08003530 con 0x7f4f28139bf0 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.6 v1:172.21.15.3:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1118396250 0 0) 0x7f4f28134ce0 con 0x7f4f2813d700 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f28073f90 con 0x7f4f2813d700 2024-05-21T17:24:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.6 v1:172.21.15.3:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4f20001b80 con 0x7f4f2813d700 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient(hunting): dump: 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2792423527 0 0) 0x7f4f2811fc90 con 0x7f4f28135c20 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 >> v1:172.21.15.203:6792/0 conn(0x7f4f28139bf0 legacy=0x7f4f281d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 >> v1:172.21.15.3:6792/0 conn(0x7f4f2813d700 legacy=0x7f4f281d41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient: _finish_hunting 0 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 monclient: found mon.a 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f28073a40 con 0x7f4f28135c20 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient: _finish_auth 0 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient: _check_auth_tickets 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 5 monclient: authenticate success, global_id 4824 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: _renew_subs 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f2811f9b0 con 0x7f4f28135c20 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: _renew_subs 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f281c27f0 con 0x7f4f28135c20 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f100038d0 con 0x7f4f28135c20 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient: handle_config config(0 keys) 2024-05-21T17:24:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3769271080 0 0) 0x7f4f10003e40 con 0x7f4f28135c20 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2559a640 4 set_mon_vals no callback set 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 10 monclient: dump: 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 4db3b12d-01c4-4d6c-b29b-1648ee37f87f 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-05-21T17:20:49.979435+0000 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.203:6789/0 mon.f 2024-05-21T17:24:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.b 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:3: v1:172.21.15.203:6790/0 mon.g 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:4: v1:172.21.15.3:6791/0 mon.c 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:5: v1:172.21.15.203:6791/0 mon.h 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:6: v1:172.21.15.3:6792/0 mon.d 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:7: v1:172.21.15.203:6792/0 mon.i 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:8: v1:172.21.15.3:6793/0 mon.e 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 5) ==== 45011+0+0 (unknown 2328380115 0 0) 0x7f4f10003fe0 con 0x7f4f28135c20 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f0c005090 con 0x7f4f28135c20 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 3726191069 0 0) 0x7f4f10045b70 con 0x7f4f28135c20 2024-05-21T17:24:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.512+0000 7f4f1ffff640 10 monclient: get_auth_request con 0x7f4f080394d0 auth_method 0 2024-05-21T17:24:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.516+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+178841 (unknown 1092875540 0 3612521739) 0x7f4f10015cd0 con 0x7f4f28135c20 2024-05-21T17:24:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.516+0000 7f4f1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-21T17:24:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.516+0000 7f4f1effd640 10 monclient: _finish_command 1 = system:0 2024-05-21T17:24:06.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.680+0000 7f4f2cd4d640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-21T17:24:06.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.680+0000 7f4f2cd4d640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-21T17:24:06.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.680+0000 7f4f2cd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-05-21T17:24:06.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.680+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 --> v1:172.21.15.3:6789/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f4f0c005380 con 0x7f4f28135c20 2024-05-21T17:24:06.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.684+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) ==== 123+0+0 (unknown 4099060384 0 0) 0x7f4f100155d0 con 0x7f4f28135c20 2024-05-21T17:24:06.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.684+0000 7f4f1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-21T17:24:06.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.684+0000 7f4f1effd640 10 monclient: _finish_command 2 = system:0 2024-05-21T17:24:06.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.688+0000 7f4f1effd640 1 -- 172.21.15.3:0/4055932324 <== mon.0 v1:172.21.15.3:6789/0 11 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f4f10015140 con 0x7f4f28135c20 2024-05-21T17:24:06.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.688+0000 7f4f1effd640 10 monclient: handle_config config(1 keys) 2024-05-21T17:24:06.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.688+0000 7f4f24d99640 4 set_mon_vals no callback set 2024-05-21T17:24:06.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.688+0000 7f4f24d99640 10 set_mon_vals mon_health_to_clog = false 2024-05-21T17:24:06.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.692+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 >> v1:172.21.15.203:6812/343827626 conn(0x7f4f080394d0 legacy=0x7f4f0803b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.692+0000 7f4f2cd4d640 10 monclient: shutdown 2024-05-21T17:24:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.692+0000 7f4f2cd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-21T17:24:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.692+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 >> v1:172.21.15.3:6789/0 conn(0x7f4f28135c20 legacy=0x7f4f281345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-21T17:24:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.692+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 shutdown_connections 2024-05-21T17:24:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.692+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 >> 172.21.15.3:0/4055932324 conn(0x7f4f280a3b90 msgr2=0x7f4f28138890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-21T17:24:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.692+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 shutdown_connections 2024-05-21T17:24:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-21T17:24:06.692+0000 7f4f2cd4d640 1 -- 172.21.15.3:0/4055932324 wait complete. 2024-05-21T17:24:06.711 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-21T17:24:06.711 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-21T17:24:06.711 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-21T17:24:06.712 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:06.776 INFO:tasks.ceph.osd.0:Stopped 2024-05-21T17:24:06.777 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-21T17:24:06.777 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:06.863 INFO:tasks.ceph.osd.1:Stopped 2024-05-21T17:24:06.863 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-21T17:24:06.863 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:06.962 INFO:tasks.ceph.osd.2:Stopped 2024-05-21T17:24:06.962 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-21T17:24:06.962 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.025 INFO:tasks.ceph.osd.3:Stopped 2024-05-21T17:24:07.025 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-21T17:24:07.025 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.112 INFO:tasks.ceph.osd.4:Stopped 2024-05-21T17:24:07.113 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-21T17:24:07.113 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.189 INFO:tasks.ceph.osd.5:Stopped 2024-05-21T17:24:07.190 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-21T17:24:07.190 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-21T17:24:07.190 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.217 INFO:tasks.ceph.mgr.x:Stopped 2024-05-21T17:24:07.217 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-21T17:24:07.217 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-21T17:24:07.217 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.241 INFO:tasks.ceph.mon.a:Stopped 2024-05-21T17:24:07.241 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-21T17:24:07.241 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.295 INFO:tasks.ceph.mon.b:Stopped 2024-05-21T17:24:07.296 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-21T17:24:07.296 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.353 INFO:tasks.ceph.mon.c:Stopped 2024-05-21T17:24:07.353 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-05-21T17:24:07.353 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.413 INFO:tasks.ceph.mon.d:Stopped 2024-05-21T17:24:07.413 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-05-21T17:24:07.413 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.477 INFO:tasks.ceph.mon.e:Stopped 2024-05-21T17:24:07.477 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-05-21T17:24:07.477 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.492 INFO:tasks.ceph.mon.f:Stopped 2024-05-21T17:24:07.493 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-05-21T17:24:07.493 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.555 INFO:tasks.ceph.mon.g:Stopped 2024-05-21T17:24:07.555 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-05-21T17:24:07.555 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.616 INFO:tasks.ceph.mon.h:Stopped 2024-05-21T17:24:07.617 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-05-21T17:24:07.617 INFO:teuthology.orchestra.run:waiting for 300 2024-05-21T17:24:07.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-05-21T17:24:07.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-05-21T17:24:07.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-05-21T17:24:07.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-05-21T17:24:07.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.d is failed for ~0s 2024-05-21T17:24:07.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.e is failed for ~0s 2024-05-21T17:24:07.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.f is failed for ~0s 2024-05-21T17:24:07.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.g is failed for ~0s 2024-05-21T17:24:07.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.h is failed for ~0s 2024-05-21T17:24:07.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.i is failed for ~0s 2024-05-21T17:24:07.676 INFO:tasks.ceph.mon.i:Stopped 2024-05-21T17:24:07.676 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-21T17:24:07.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | head -n 1 2024-05-21T17:24:07.694 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:24:07.694 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-21T17:24:08.007 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:24:08.007 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-21T17:24:08.209 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:24:08.209 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-21T17:24:08.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:24:08.399 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-21T17:24:08.689 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:24:08.689 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-21T17:24:08.884 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:24:08.885 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-21T17:24:09.042 INFO:tasks.ceph:Archiving mon data... 2024-05-21T17:24:09.043 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.a.tgz 2024-05-21T17:24:09.043 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-05-21T17:24:09.049 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.ahCqqSTYhi 2024-05-21T17:24:09.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ahCqqSTYhi 2024-05-21T17:24:09.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.ahCqqSTYhi 2024-05-21T17:24:09.232 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.ahCqqSTYhi is 106KB 2024-05-21T17:24:09.248 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.ahCqqSTYhi 2024-05-21T17:24:09.253 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.b.tgz 2024-05-21T17:24:09.253 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-05-21T17:24:09.299 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.7xfYxHoXnG 2024-05-21T17:24:09.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.7xfYxHoXnG 2024-05-21T17:24:09.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.7xfYxHoXnG 2024-05-21T17:24:09.484 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.7xfYxHoXnG is 104KB 2024-05-21T17:24:09.538 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.7xfYxHoXnG 2024-05-21T17:24:09.543 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.c.tgz 2024-05-21T17:24:09.544 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-05-21T17:24:09.591 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.tzgpASeR2d 2024-05-21T17:24:09.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.tzgpASeR2d 2024-05-21T17:24:09.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.tzgpASeR2d 2024-05-21T17:24:09.776 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.tzgpASeR2d is 106KB 2024-05-21T17:24:09.792 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.tzgpASeR2d 2024-05-21T17:24:09.797 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.d.tgz 2024-05-21T17:24:09.797 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-05-21T17:24:09.843 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.INN7YAU6FW 2024-05-21T17:24:09.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.INN7YAU6FW 2024-05-21T17:24:09.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.INN7YAU6FW 2024-05-21T17:24:10.027 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.INN7YAU6FW is 106KB 2024-05-21T17:24:10.042 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.INN7YAU6FW 2024-05-21T17:24:10.047 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.e.tgz 2024-05-21T17:24:10.047 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-05-21T17:24:10.095 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.2hW7VzzczL 2024-05-21T17:24:10.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.2hW7VzzczL 2024-05-21T17:24:10.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.2hW7VzzczL 2024-05-21T17:24:10.280 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.2hW7VzzczL is 106KB 2024-05-21T17:24:10.295 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.2hW7VzzczL 2024-05-21T17:24:10.300 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.f.tgz 2024-05-21T17:24:10.300 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-05-21T17:24:10.306 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.xWPQ4rJr2I 2024-05-21T17:24:10.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.xWPQ4rJr2I 2024-05-21T17:24:10.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.xWPQ4rJr2I 2024-05-21T17:24:10.494 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.xWPQ4rJr2I is 104KB 2024-05-21T17:24:10.510 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.xWPQ4rJr2I 2024-05-21T17:24:10.515 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.g.tgz 2024-05-21T17:24:10.515 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-05-21T17:24:10.562 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.rJs2ynHzD9 2024-05-21T17:24:10.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.rJs2ynHzD9 2024-05-21T17:24:10.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.rJs2ynHzD9 2024-05-21T17:24:10.748 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.rJs2ynHzD9 is 106KB 2024-05-21T17:24:10.764 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.rJs2ynHzD9 2024-05-21T17:24:10.769 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.h.tgz 2024-05-21T17:24:10.769 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-05-21T17:24:10.818 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.aBiYYdmWUA 2024-05-21T17:24:10.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.aBiYYdmWUA 2024-05-21T17:24:10.898 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.aBiYYdmWUA 2024-05-21T17:24:11.007 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.aBiYYdmWUA is 106KB 2024-05-21T17:24:11.021 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.aBiYYdmWUA 2024-05-21T17:24:11.027 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/data/mon.i.tgz 2024-05-21T17:24:11.027 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-05-21T17:24:11.073 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.X5DTpw0jt3 2024-05-21T17:24:11.074 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.X5DTpw0jt3 2024-05-21T17:24:11.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.X5DTpw0jt3 2024-05-21T17:24:11.258 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.X5DTpw0jt3 is 106KB 2024-05-21T17:24:11.274 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.X5DTpw0jt3 2024-05-21T17:24:11.280 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-21T17:24:11.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-21T17:24:11.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-21T17:24:11.343 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-21T17:24:11.382 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-21T17:24:11.382 INFO:tasks.ceph:Archiving crash dumps... 2024-05-21T17:24:11.384 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/remote/smithi003/crash 2024-05-21T17:24:11.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-21T17:24:11.400 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/remote/smithi203/crash 2024-05-21T17:24:11.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-21T17:24:11.436 INFO:tasks.ceph:Compressing logs... 2024-05-21T17:24:11.437 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-21T17:24:11.451 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-21T17:24:11.468 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28880.log 2024-05-21T17:24:11.470 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28802.log 2024-05-21T17:24:11.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27207.log 2024-05-21T17:24:11.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28029.log 2024-05-21T17:24:11.470 INFO:teuthology.orchestra.run.smithi003.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28880.log.gz 2024-05-21T17:24:11.470 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27007.log 2024-05-21T17:24:11.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27207.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28802.log.gz 2024-05-21T17:24:11.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27847.log 2024-05-21T17:24:11.471 INFO:teuthology.orchestra.run.smithi003.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27207.log.gz 2024-05-21T17:24:11.471 INFO:teuthology.orchestra.run.smithi003.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28029.log.gz 2024-05-21T17:24:11.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27007.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20226.log 2024-05-21T17:24:11.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27847.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27007.log.gz 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28055.log 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.tmp-client.admin.20226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27743.log 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20226.log.gz 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27847.log.gz 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28055.log: /var/log/ceph/ceph-client.admin.27743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27058.log 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28055.log.gz 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28542.log 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27743.log.gz 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27208.log 2024-05-21T17:24:11.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29010.log 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27058.log.gz 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28698.log 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28542.log.gz 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27208.log.gz 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27214.log 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29010.log.gz 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28081.log 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23909.log 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28698.log.gz 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28081.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27214.log.gz 2024-05-21T17:24:11.476 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24221.log 2024-05-21T17:24:11.479 INFO:teuthology.orchestra.run.smithi003.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28081.log.gz 2024-05-21T17:24:11.479 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.23909.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.24221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28750.log 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr: 88.4%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.23909.log.gz 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27053.log 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.24221.log.gz 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28776.log 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28750.log.gz 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26957.log 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28333.log 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28776.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27053.log.gz 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.26957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28672.log 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28776.log.gz 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.26957.log.gz 2024-05-21T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28333.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-21T17:24:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28133.log 2024-05-21T17:24:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28672.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29034.log 2024-05-21T17:24:11.481 INFO:teuthology.orchestra.run.smithi003.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28333.log.gz 2024-05-21T17:24:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.e.log 2024-05-21T17:24:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28133.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28672.log.gz 2024-05-21T17:24:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27507.log 2024-05-21T17:24:11.481 INFO:teuthology.orchestra.run.smithi003.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28133.log.gz 2024-05-21T17:24:11.483 INFO:teuthology.orchestra.run.smithi003.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29034.log.gz 2024-05-21T17:24:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27769.log 2024-05-21T17:24:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28906.log 2024-05-21T17:24:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27507.log: /var/log/ceph/ceph-client.admin.27769.log: /var/log/ceph/ceph-client.admin.28906.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27507.log.gz 2024-05-21T17:24:11.483 INFO:teuthology.orchestra.run.smithi003.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27769.log.gz 2024-05-21T17:24:11.483 INFO:teuthology.orchestra.run.smithi003.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28906.log.gz 2024-05-21T17:24:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28620.log 2024-05-21T17:24:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28724.log 2024-05-21T17:24:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24147.log 2024-05-21T17:24:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28724.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28620.log.gz 2024-05-21T17:24:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28516.log 2024-05-21T17:24:11.486 INFO:teuthology.orchestra.run.smithi003.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28724.log.gz 2024-05-21T17:24:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.24147.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.24147.log.gz 2024-05-21T17:24:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28958.log 2024-05-21T17:24:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26913.log 2024-05-21T17:24:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28984.log 2024-05-21T17:24:11.486 INFO:teuthology.orchestra.run.smithi003.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28516.log.gz 2024-05-21T17:24:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.26913.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28958.log.gz 2024-05-21T17:24:11.487 INFO:teuthology.orchestra.run.smithi003.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.26913.log.gz 2024-05-21T17:24:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27388.log 2024-05-21T17:24:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28412.log 2024-05-21T17:24:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27388.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28984.log.gz -5 2024-05-21T17:24:11.488 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.28307.log 2024-05-21T17:24:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28412.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27388.log.gz 2024-05-21T17:24:11.489 INFO:teuthology.orchestra.run.smithi003.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28412.log.gz 2024-05-21T17:24:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27195.log 2024-05-21T17:24:11.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27351.log 2024-05-21T17:24:11.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27899.log 2024-05-21T17:24:11.491 INFO:teuthology.orchestra.run.smithi003.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28307.log.gz 2024-05-21T17:24:11.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27351.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27195.log.gz 2024-05-21T17:24:11.491 INFO:teuthology.orchestra.run.smithi003.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27351.log.gz 2024-05-21T17:24:11.491 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28385.log 2024-05-21T17:24:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27611.log 2024-05-21T17:24:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27055.log 2024-05-21T17:24:11.493 INFO:teuthology.orchestra.run.smithi003.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27899.log.gz 2024-05-21T17:24:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27611.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28385.log.gz 2024-05-21T17:24:11.493 INFO:teuthology.orchestra.run.smithi003.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27611.log.gz 2024-05-21T17:24:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28107.log 2024-05-21T17:24:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28828.log 2024-05-21T17:24:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28107.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 87.1% 2024-05-21T17:24:11.495 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27055.log.gz 2024-05-21T17:24:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28828.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28107.log.gz 2024-05-21T17:24:11.495 INFO:teuthology.orchestra.run.smithi003.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28828.log.gz 2024-05-21T17:24:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.d.log 2024-05-21T17:24:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27360.log 2024-05-21T17:24:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27585.log 2024-05-21T17:24:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27360.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27360.log.gz 2024-05-21T17:24:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27977.log 2024-05-21T17:24:11.499 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27585.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27585.log.gz 2024-05-21T17:24:11.499 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28187.log 2024-05-21T17:24:11.500 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27977.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.27977.log.gz 2024-05-21T17:24:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28594.log 2024-05-21T17:24:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28187.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28187.log.gz 2024-05-21T17:24:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28003.log 2024-05-21T17:24:11.503 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28359.log 2024-05-21T17:24:11.504 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28594.log: /var/log/ceph/ceph-client.admin.28003.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28594.log.gz 2024-05-21T17:24:11.504 INFO:teuthology.orchestra.run.smithi003.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28003.log.gz 2024-05-21T17:24:11.504 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-21T17:24:11.505 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27795.log 2024-05-21T17:24:11.506 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28359.log: /var/log/ceph/ceph.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28359.log.gz 2024-05-21T17:24:11.506 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27663.log 2024-05-21T17:24:11.507 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24235.log 2024-05-21T17:24:11.507 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.f.log 2024-05-21T17:24:11.507 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27017.log 2024-05-21T17:24:11.508 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.f.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24235.log.gz 2024-05-21T17:24:11.508 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26982.log 2024-05-21T17:24:11.508 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-21T17:24:11.508 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.27017.log: /var/log/ceph/ceph-client.admin.26982.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-21T17:24:11.508 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27018.log 2024-05-21T17:24:11.510 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.i.log 2024-05-21T17:24:11.510 INFO:teuthology.orchestra.run.smithi203.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27017.log.gz 2024-05-21T17:24:11.510 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.27018.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.g.log 2024-05-21T17:24:11.510 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.i.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-21T17:24:11.510 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.g.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-21T17:24:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27795.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27795.log.gz 2024-05-21T17:24:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27533.log 2024-05-21T17:24:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27663.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27663.log.gz 2024-05-21T17:24:11.512 INFO:teuthology.orchestra.run.smithi003.stderr: 95.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-21T17:24:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28646.log 2024-05-21T17:24:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-21T17:24:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27533.log: /var/log/ceph/ceph-client.admin.28646.log: 87.1% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27533.log.gz 2024-05-21T17:24:11.512 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28646.log.gz 2024-05-21T17:24:11.513 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27637.log 2024-05-21T17:24:11.513 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28854.log 2024-05-21T17:24:11.513 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.27637.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27637.log.gz 2024-05-21T17:24:11.514 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.26982.log.gz 2024-05-21T17:24:11.514 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24263.log 2024-05-21T17:24:11.514 INFO:teuthology.orchestra.run.smithi203.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.27018.log.gz 2024-05-21T17:24:11.514 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.h.log 2024-05-21T17:24:11.514 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24263.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24263.log.gz 2024-05-21T17:24:11.516 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27821.log 2024-05-21T17:24:11.516 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28854.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28854.log.gz 2024-05-21T17:24:11.516 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-21T17:24:11.516 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.h.log: 94.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-21T17:24:11.517 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24287.log 2024-05-21T17:24:11.522 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: 96.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-21T17:24:11.523 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24187.log 2024-05-21T17:24:11.523 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24287.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24287.log.gz 2024-05-21T17:24:11.523 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-21T17:24:11.524 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24187.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24187.log.gz 2024-05-21T17:24:11.524 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27016.log 2024-05-21T17:24:11.526 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27355.log 2024-05-21T17:24:11.528 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27821.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27821.log.gz 2024-05-21T17:24:11.530 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28568.log 2024-05-21T17:24:11.531 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27355.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27355.log.gz 2024-05-21T17:24:11.537 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-21T17:24:11.538 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26983.log 2024-05-21T17:24:11.541 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.27016.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.27016.log.gz 2024-05-21T17:24:11.545 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28568.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28568.log.gz 2024-05-21T17:24:11.547 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27059.log 2024-05-21T17:24:11.551 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.audit.log: 96.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-21T17:24:11.553 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24311.log 2024-05-21T17:24:11.554 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.26983.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26983.log.gz 2024-05-21T17:24:11.560 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27391.log 2024-05-21T17:24:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27059.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27059.log.gz 2024-05-21T17:24:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28438.log 2024-05-21T17:24:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27391.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27391.log.gz 2024-05-21T17:24:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27689.log 2024-05-21T17:24:11.564 INFO:teuthology.orchestra.run.smithi203.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-21T17:24:11.565 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24211.log 2024-05-21T17:24:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28438.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28438.log.gz 2024-05-21T17:24:11.566 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24311.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.24311.log.gz 2024-05-21T17:24:11.567 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24211.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.24211.log.gz 2024-05-21T17:24:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27210.log 2024-05-21T17:24:11.574 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27689.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.27689.log.gz 2024-05-21T17:24:11.575 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28281.log 2024-05-21T17:24:11.575 INFO:teuthology.orchestra.run.smithi203.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.f.log.gz 2024-05-21T17:24:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27210.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27210.log.gz 2024-05-21T17:24:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27048.log 2024-05-21T17:24:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28281.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28281.log.gz 2024-05-21T17:24:11.582 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28161.log 2024-05-21T17:24:11.591 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27048.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27048.log.gz 2024-05-21T17:24:11.598 INFO:teuthology.orchestra.run.smithi003.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-21T17:24:11.598 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28216.log 2024-05-21T17:24:11.598 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28161.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28161.log.gz 2024-05-21T17:24:11.598 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27387.log 2024-05-21T17:24:11.598 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28216.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28216.log.gz 2024-05-21T17:24:11.598 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28490.log 2024-05-21T17:24:11.599 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27387.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27387.log.gz 2024-05-21T17:24:11.599 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27559.log 2024-05-21T17:24:11.600 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28490.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28490.log.gz 2024-05-21T17:24:11.600 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27925.log 2024-05-21T17:24:11.601 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27559.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27559.log.gz 2024-05-21T17:24:11.602 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28464.log 2024-05-21T17:24:11.602 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27925.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27925.log.gz 2024-05-21T17:24:11.603 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26859.log 2024-05-21T17:24:11.604 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28464.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28464.log.gz 2024-05-21T17:24:11.604 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27046.log 2024-05-21T17:24:11.605 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.26859.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.26859.log.gz 2024-05-21T17:24:11.605 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-21T17:24:11.606 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27046.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27046.log.gz 2024-05-21T17:24:11.607 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28932.log 2024-05-21T17:24:11.620 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.b.log: 90.3% -- replaced with /var/log/ceph/ceph-mon.e.log.gz 2024-05-21T17:24:11.622 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27715.log 2024-05-21T17:24:11.623 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.28932.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28932.log.gz 2024-05-21T17:24:11.624 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27215.log 2024-05-21T17:24:11.624 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26887.log 2024-05-21T17:24:11.625 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27215.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27715.log.gz 2024-05-21T17:24:11.625 INFO:teuthology.orchestra.run.smithi003.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27215.log.gz 2024-05-21T17:24:11.625 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27873.log 2024-05-21T17:24:11.626 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.26887.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26887.log.gz 2024-05-21T17:24:11.627 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27951.log 2024-05-21T17:24:11.628 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27873.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27873.log.gz 2024-05-21T17:24:11.628 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26834.log 2024-05-21T17:24:11.629 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.27951.log: /var/log/ceph/ceph-client.admin.26834.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27951.log.gz 2024-05-21T17:24:11.629 INFO:teuthology.orchestra.run.smithi003.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26834.log.gz 2024-05-21T17:24:11.631 INFO:teuthology.orchestra.run.smithi003.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.d.log.gz 2024-05-21T17:24:11.649 INFO:teuthology.orchestra.run.smithi003.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-21T17:24:11.655 INFO:teuthology.orchestra.run.smithi203.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.g.log.gz 2024-05-21T17:24:11.661 INFO:teuthology.orchestra.run.smithi203.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.h.log.gz 2024-05-21T17:24:11.663 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-21T17:24:11.667 INFO:teuthology.orchestra.run.smithi003.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-21T17:24:11.675 INFO:teuthology.orchestra.run.smithi203.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.i.log.gz 2024-05-21T17:24:11.697 INFO:teuthology.orchestra.run.smithi203.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-21T17:24:11.771 INFO:teuthology.orchestra.run.smithi003.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-21T17:24:12.042 INFO:teuthology.orchestra.run.smithi003.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-21T17:24:12.075 INFO:teuthology.orchestra.run.smithi203.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-21T17:24:12.076 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-05-21T17:24:12.076 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m0.592s 2024-05-21T17:24:12.076 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m1.448s 2024-05-21T17:24:12.077 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.084s 2024-05-21T17:24:12.083 INFO:teuthology.orchestra.run.smithi003.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-21T17:24:12.084 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-05-21T17:24:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m0.631s 2024-05-21T17:24:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m2.106s 2024-05-21T17:24:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.082s 2024-05-21T17:24:12.085 INFO:tasks.ceph:Archiving logs... 2024-05-21T17:24:12.085 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/remote/smithi003/log 2024-05-21T17:24:12.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-21T17:24:12.422 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/remote/smithi203/log 2024-05-21T17:24:12.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-21T17:24:12.604 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-21T17:24:12.613 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-21T17:24:12.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-21T17:24:12.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.d has been restored 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.e has been restored 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.f has been restored 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.g has been restored 2024-05-21T17:24:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.h has been restored 2024-05-21T17:24:12.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.i has been restored 2024-05-21T17:24:12.697 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-21T17:24:12.705 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-21T17:24:12.706 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-21T17:24:12.708 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-21T17:24:12.720 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-05-21T17:24:12.720 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-05-21T17:24:12.720 INFO:teuthology.orchestra.run.smithi203.stdout:*hv01.front.sepi 67.205.162.81 3 u 51 64 37 0.109 -1.512 1.040 2024-05-21T17:24:12.720 INFO:teuthology.orchestra.run.smithi203.stdout:+hv02.front.sepi 63.231.80.2 3 u 48 64 37 0.078 -1.185 1.063 2024-05-21T17:24:12.720 INFO:teuthology.orchestra.run.smithi203.stdout:+hv03.front.sepi 74.6.168.72 3 u 43 64 37 0.135 -3.154 0.986 2024-05-21T17:24:12.720 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:24:12.721 INFO:teuthology.orchestra.run.smithi003.stdout: remote refid st t when poll reach delay offset jitter 2024-05-21T17:24:12.721 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================== 2024-05-21T17:24:12.721 INFO:teuthology.orchestra.run.smithi003.stdout:*hv01.front.sepi 67.205.162.81 3 u 50 64 37 0.119 -0.076 1.703 2024-05-21T17:24:12.721 INFO:teuthology.orchestra.run.smithi003.stdout:+hv02.front.sepi 63.231.80.2 3 u 48 64 37 0.085 +1.280 0.729 2024-05-21T17:24:12.722 INFO:teuthology.orchestra.run.smithi003.stdout:+hv03.front.sepi 74.6.168.72 3 u 46 64 37 0.069 -0.507 0.857 2024-05-21T17:24:12.722 INFO:teuthology.orchestra.run.smithi003.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-21T17:24:12.722 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-21T17:24:12.731 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-21T17:24:12.731 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-21T17:24:12.765 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-21T17:24:12.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-21T17:24:12.831 INFO:teuthology.task.internal:Duration was 748.744696 seconds 2024-05-21T17:24:12.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-21T17:24:12.865 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-21T17:24:12.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-21T17:24:12.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-21T17:24:12.905 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-21T17:24:12.905 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-05-21T17:24:12.905 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-21T17:24:12.961 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-05-21T17:24:12.961 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-21T17:24:12.971 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-21T17:24:12.971 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-21T17:24:13.004 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-21T17:24:13.037 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-21T17:24:13.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-21T17:24:13.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-21T17:24:13.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-21T17:24:13.145 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-21T17:24:13.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-21T17:24:13.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-21T17:24:13.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-21T17:24:13.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-21T17:24:13.208 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-21T17:24:13.216 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-05-21T17:24:13.221 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-05-21T17:24:13.247 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-21T17:24:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:24:13.291 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-21T17:24:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T17:24:13.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-21T17:24:13.304 INFO:teuthology.task.internal:Transferring archived files... 2024-05-21T17:24:13.305 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/remote/smithi003 2024-05-21T17:24:13.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-21T17:24:13.355 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-20_20:45:46-rados-wip-yuri3-testing-2024-05-20-1050-distro-default-smithi/7716922/remote/smithi203 2024-05-21T17:24:13.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-21T17:24:13.378 INFO:teuthology.task.internal:Removing archive directory... 2024-05-21T17:24:13.378 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-21T17:24:13.396 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-21T17:24:13.422 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-21T17:24:13.435 INFO:teuthology.task.internal:Not uploading archives. 2024-05-21T17:24:13.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-21T17:24:13.457 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-21T17:24:13.457 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-21T17:24:13.460 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-21T17:24:13.463 INFO:teuthology.orchestra.run.smithi003.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 21 17:24 /home/ubuntu/cephtest 2024-05-21T17:24:13.470 INFO:teuthology.orchestra.run.smithi203.stdout: 658466 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 21 17:24 /home/ubuntu/cephtest 2024-05-21T17:24:13.472 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-21T17:24:13.491 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-21T17:24:13.559 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} duration: 748.7446956634521 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-05-21T17:24:13.559 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-21T17:24:13.643 INFO:teuthology.run:pass